2024-04-23T17:58:58.854 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T17:58:58.855 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100 --name leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/orig.config.yaml 2024-04-23T17:58:58.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T17:58:59.009 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7670100' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5182 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2MeWAIPf5+honJbPW8q95+kLsIwe+CaCOIRqW2Y7Aod7GOU66CeYemrJfizi3EtDpwpcaT8Excx+djS4aY/9Q= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWASL7n+ZS0ePdXEdYuixB6KXomVNynUNGx5YgwJcztvQFIEm0VWH6L/xg0Oby2lZ28u1KTD8VD9LSNzCgR6tY= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEXOMv1VMbXuz8T+ca7x7Q46YAkwk+kwEoErjZOj4y2xWqW/D8IqQwSISRgiXJa0EQGmp7flPID8x+TQzcQayk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_17:44:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T17:58:59.009 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa; will attempt to use it 2024-04-23T17:58:59.010 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks 2024-04-23T17:58:59.010 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T17:58:59.012 INFO:teuthology.task.internal:Checking packages... 2024-04-23T17:58:59.036 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-23T17:58:59.036 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T17:58:59.036 INFO:teuthology.packaging:ref: None 2024-04-23T17:58:59.036 INFO:teuthology.packaging:tag: None 2024-04-23T17:58:59.036 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-23T17:58:59.036 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T17:58:59.037 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-23T17:58:59.289 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-23T17:58:59.291 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T17:58:59.301 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T17:58:59.301 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T17:58:59.308 INFO:teuthology.task.internal:Saving configuration 2024-04-23T17:58:59.330 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T17:58:59.336 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T17:58:59.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.432392', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2MeWAIPf5+honJbPW8q95+kLsIwe+CaCOIRqW2Y7Aod7GOU66CeYemrJfizi3EtDpwpcaT8Excx+djS4aY/9Q='} 2024-04-23T17:58:59.382 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.433806', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWASL7n+ZS0ePdXEdYuixB6KXomVNynUNGx5YgwJcztvQFIEm0VWH6L/xg0Oby2lZ28u1KTD8VD9LSNzCgR6tY='} 2024-04-23T17:58:59.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.435509', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEXOMv1VMbXuz8T+ca7x7Q46YAkwk+kwEoErjZOj4y2xWqW/D8IqQwSISRgiXJa0EQGmp7flPID8x+TQzcQayk='} 2024-04-23T17:58:59.407 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T17:58:59.412 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-23T17:58:59.412 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-23T17:58:59.412 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-23T17:58:59.412 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T17:58:59.524 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f24ec95c0d0>, signals=[15]) 2024-04-23T17:58:59.524 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T17:58:59.530 INFO:teuthology.task.internal:Opening connections... 2024-04-23T17:58:59.530 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T17:58:59.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:58:59.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-04-23T17:58:59.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:58:59.687 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-04-23T17:58:59.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T17:58:59.761 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T17:58:59.767 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-04-23T17:58:59.786 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-04-23T17:58:59.786 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-04-23T17:58:59.844 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:58:59.845 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:58:59.846 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-04-23T17:58:59.885 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-04-23T17:58:59.902 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-04-23T17:58:59.903 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-04-23T17:58:59.960 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-04-23T17:58:59.960 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-04-23T17:58:59.960 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-04-23T17:58:59.960 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:58:59.961 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:58:59.962 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-04-23T17:58:59.992 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-04-23T17:59:00.010 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-04-23T17:59:00.010 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-04-23T17:59:00.068 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-04-23T17:59:00.068 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-04-23T17:59:00.068 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-04-23T17:59:00.068 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T17:59:00.069 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T17:59:00.069 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-04-23T17:59:00.092 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T17:59:00.102 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T17:59:00.107 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T17:59:00.108 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:59:00.110 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:59:00.113 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-04-23T17:59:00.129 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T17:59:00.134 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T17:59:00.135 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:59:00.169 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:59:00.172 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-04-23T17:59:00.256 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T17:59:00.341 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T17:59:00.379 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-23T17:59:00.380 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-23T17:59:00.380 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-23T17:59:00.380 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:59:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:00.398 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-04-23T17:59:00.457 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-437.el9.x86_64 2024-04-23T17:59:00.458 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-437.el9.x86_64 2024-04-23T17:59:00.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-04-23T17:59:04.114 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-23T17:59:05.915 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-23T17:59:10.998 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-04-23T17:59:12.808 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-23T17:59:14.245 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-23T17:59:19.542 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-23T17:59:21.431 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T17:59:21.432 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T17:59:21.523 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-23T17:59:21.524 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-23T17:59:21.524 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-23T17:59:21.641 DEBUG:teuthology.orchestra.run.smithi026:> echo no | sudo yum reinstall kernel || true 2024-04-23T17:59:22.160 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 05:59:19 PM UTC. 2024-04-23T17:59:22.288 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-23T17:59:22.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.305 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-23T17:59:22.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.305 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalling: 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 255 k 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 0 2024-04-23T17:59:22.306 INFO:teuthology.orchestra.run.smithi026.stderr:Operation aborted. 2024-04-23T17:59:22.355 INFO:teuthology.orchestra.run.smithi026.stdout:Is this ok [y/N]: 2024-04-23T17:59:22.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum reinstall -y kernel || true 2024-04-23T17:59:22.869 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 05:59:19 PM UTC. 2024-04-23T17:59:22.990 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalling: 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 255 k 2024-04-23T17:59:22.991 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 0 2024-04-23T17:59:22.992 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-23T17:59:23.660 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-437.el9.x86_64.rpm 421 kB/s | 255 kB 00:00 2024-04-23T17:59:23.661 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-23T17:59:23.661 INFO:teuthology.orchestra.run.smithi026.stdout:Total 381 kB/s | 255 kB 00:00 2024-04-23T17:59:23.682 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-23T17:59:23.864 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-23T17:59:23.864 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-23T17:59:23.870 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-23T17:59:23.870 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-23T17:59:23.999 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-23T17:59:24.081 INFO:teuthology.orchestra.run.smithi026.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T17:59:24.082 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:59:24.728 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:59:24.728 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalled: 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T17:59:25.397 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-23T17:59:25.574 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T17:59:25.622 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T17:59:25.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T17:59:25.622 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T17:59:25.623 DEBUG:teuthology.parallel:result is None 2024-04-23T17:59:25.623 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T17:59:25.630 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T17:59:25.630 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:59:25.633 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:59:25.635 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T17:59:25.653 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T17:59:25.659 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T17:59:25.669 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T17:59:25.669 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:59:25.688 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:59:25.693 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T17:59:25.805 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T17:59:25.812 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T17:59:25.813 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:59:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:25.829 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T17:59:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:25.846 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:59:25.848 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:59:25.873 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T17:59:25.906 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.912 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.926 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.928 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.938 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.948 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T17:59:25.950 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T17:59:25.957 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T17:59:25.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:59:25.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:59:25.993 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T17:59:26.019 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T17:59:26.029 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T17:59:26.029 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:59:26.031 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:59:26.062 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T17:59:26.077 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.177 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.298 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T17:59:26.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:59:26.362 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.404 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.451 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.530 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.576 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T17:59:26.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:59:26.641 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.679 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T17:59:26.737 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.820 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T17:59:26.862 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T17:59:26.862 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T17:59:26.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-04-23T17:59:26.931 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-04-23T17:59:26.933 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-04-23T17:59:26.966 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:59:26.978 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:59:27.008 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:59:27.477 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T17:59:27.485 INFO:teuthology.task.internal:Starting timer... 2024-04-23T17:59:27.485 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T17:59:27.496 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T17:59:27.505 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-23T17:59:27.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-04-23T17:59:27.559 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-04-23T17:59:27.560 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-04-23T17:59:27.617 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-04-23T17:59:27.619 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-04-23T17:59:27.688 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-04-23T17:59:27.690 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T17:59:27.691 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-04-23T17:59:27.721 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-04-23T17:59:27.722 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-04-23T17:59:27.750 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-04-23T17:59:27.750 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-04-23T17:59:27.784 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-04-23T17:59:27.784 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive'} 2024-04-23T17:59:27.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:59:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:27.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:59:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:27.837 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T17:59:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:59:27.864 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T17:59:27.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:59:27.899 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:59:27.932 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-04-23T17:59:27.958 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T17:59:27.967 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T17:59:27.973 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T17:59:28.106 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T17:59:28.121 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T17:59:28.122 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi110.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T18:03:46.157 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2024-04-23T18:03:46.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-23T18:03:46.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:03:46.240 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-04-23T18:03:46.314 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-23T18:03:46.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-23T18:03:46.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:03:46.390 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-23T18:03:46.465 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-23T18:03:46.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-23T18:03:46.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:03:46.553 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-04-23T18:03:46.631 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-04-23T18:03:46.632 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T18:03:46.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T18:03:46.647 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:03:46.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:03:46.650 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:03:46.650 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:03:46.652 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:03:46.652 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:03:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:03:46.679 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:03:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:03:46.692 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:03:46.702 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:03:46.716 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-04-23T18:03:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-04-23T18:03:46.717 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:03:46.725 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-04-23T18:03:46.726 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-04-23T18:03:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:03:46.740 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:03:46.741 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-04-23T18:03:46.751 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:03:46.752 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-04-23T18:03:46.754 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:03:46.766 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:03:46.779 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:03:46.803 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.807 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.811 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:03:46.815 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:03:46.815 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-23T18:03:46.816 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.816 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.816 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.816 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.841 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.844 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:03:46.846 INFO:teuthology.run_tasks:Running task install... 2024-04-23T18:03:46.854 DEBUG:teuthology.task.install:project ceph 2024-04-23T18:03:46.854 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-23T18:03:46.854 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-23T18:03:46.854 INFO:teuthology.task.install:Using flavor: default 2024-04-23T18:03:46.860 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T18:03:46.860 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T18:03:46.860 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:03:46.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:46.863 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:03:46.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:46.864 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:03:46.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:03:47.023 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:03:47.024 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:03:47.099 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:03:47.099 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:03:47.115 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:03:47.115 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:03:47.137 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:03:47.137 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:03:47.137 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:03:47.170 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:03:47.170 DEBUG:teuthology.orchestra.run.smithi131:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:03:47.235 DEBUG:teuthology.orchestra.run.smithi131:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:03:47.271 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:03:47.271 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:03:47.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:03:47.284 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:03:47.285 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:03:47.285 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:03:47.298 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:03:47.298 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:03:47.308 DEBUG:teuthology.orchestra.run.smithi131:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:03:47.309 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:03:47.309 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:03:47.366 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:03:47.371 INFO:teuthology.orchestra.run.smithi131.stdout:check_obsoletes = 1 2024-04-23T18:03:47.373 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum clean all 2024-04-23T18:03:47.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:03:47.443 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:03:47.447 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:03:47.508 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-04-23T18:03:47.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-04-23T18:03:47.510 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-04-23T18:03:47.511 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-04-23T18:03:47.596 INFO:teuthology.orchestra.run.smithi131.stdout:48 files removed 2024-04-23T18:03:47.624 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:03:47.735 INFO:teuthology.orchestra.run.smithi110.stdout:48 files removed 2024-04-23T18:03:47.745 INFO:teuthology.orchestra.run.smithi026.stdout:48 files removed 2024-04-23T18:03:47.762 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:03:47.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:03:48.192 INFO:teuthology.orchestra.run.smithi131.stdout:ceph packages for x86_64 269 kB/s | 83 kB 00:00 2024-04-23T18:03:48.406 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 232 kB/s | 83 kB 00:00 2024-04-23T18:03:48.435 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-23T18:03:48.542 INFO:teuthology.orchestra.run.smithi131.stdout:ceph noarch packages 38 kB/s | 12 kB 00:00 2024-04-23T18:03:48.794 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-23T18:03:48.856 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-23T18:03:48.884 INFO:teuthology.orchestra.run.smithi131.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-23T18:03:49.111 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-04-23T18:03:49.232 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-23T18:03:49.770 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-04-23T18:03:50.665 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-04-23T18:03:51.102 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-23T18:03:52.650 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-23T18:03:52.745 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-04-23T18:03:57.055 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-23T18:03:58.466 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-23T18:04:00.487 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-23T18:04:04.099 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 817 kB/s | 6.2 MB 00:07 2024-04-23T18:04:05.468 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-23T18:04:05.751 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-23T18:04:07.487 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T18:04:07.569 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:07.569 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:07.580 INFO:teuthology.orchestra.run.smithi110.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:04:07.581 INFO:teuthology.orchestra.run.smithi110.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:04:07.582 INFO:teuthology.orchestra.run.smithi110.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:04:07.583 INFO:teuthology.orchestra.run.smithi110.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:04:07.584 INFO:teuthology.orchestra.run.smithi110.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:04:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:04:07.587 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:04:07.649 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:04:07.670 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:04:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:04:07.672 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:04:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:04:07.674 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:07.675 INFO:teuthology.orchestra.run.smithi110.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:04:07.676 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:04:07.677 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:04:07.678 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:04:07.679 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:04:07.680 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:04:07.681 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:04:07.682 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:04:07.683 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:04:07.684 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:04:07.685 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-23T18:04:07.686 INFO:teuthology.orchestra.run.smithi110.stdout:Install 148 Packages 2024-04-23T18:04:07.687 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 4 Packages 2024-04-23T18:04:07.687 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:07.687 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 222 M 2024-04-23T18:04:07.687 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-04-23T18:04:08.586 INFO:teuthology.orchestra.run.smithi110.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-23T18:04:09.003 INFO:teuthology.orchestra.run.smithi110.stdout:(2/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 8.2 MB/s | 5.4 MB 00:00 2024-04-23T18:04:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.4 MB/s | 145 kB 00:00 2024-04-23T18:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 9.8 MB/s | 2.4 MB 00:00 2024-04-23T18:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout:(5/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 890 kB 00:00 2024-04-23T18:04:09.462 INFO:teuthology.orchestra.run.smithi110.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 9.4 MB/s | 1.5 MB 00:00 2024-04-23T18:04:10.145 INFO:teuthology.orchestra.run.smithi110.stdout:(7/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 12 MB/s | 21 MB 00:01 2024-04-23T18:04:10.862 INFO:teuthology.orchestra.run.smithi110.stdout:(8/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 12 MB/s | 17 MB 00:01 2024-04-23T18:04:10.904 INFO:teuthology.orchestra.run.smithi110.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 606 kB/s | 25 kB 00:00 2024-04-23T18:04:11.138 INFO:teuthology.orchestra.run.smithi110.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 11 MB/s | 11 MB 00:00 2024-04-23T18:04:11.205 INFO:teuthology.orchestra.run.smithi110.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 3.3 MB/s | 225 kB 00:00 2024-04-23T18:04:11.263 INFO:teuthology.orchestra.run.smithi110.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 464 kB/s | 27 kB 00:00 2024-04-23T18:04:11.270 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 952 kB/s | 19 MB 00:20 2024-04-23T18:04:11.322 INFO:teuthology.orchestra.run.smithi110.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 543 kB/s | 32 kB 00:00 2024-04-23T18:04:11.439 INFO:teuthology.orchestra.run.smithi110.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 6.2 MB/s | 747 kB 00:00 2024-04-23T18:04:11.506 INFO:teuthology.orchestra.run.smithi110.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.4 MB/s | 163 kB 00:00 2024-04-23T18:04:11.565 INFO:teuthology.orchestra.run.smithi110.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 2.1 MB/s | 126 kB 00:00 2024-04-23T18:04:11.648 INFO:teuthology.orchestra.run.smithi110.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 5.9 MB/s | 500 kB 00:00 2024-04-23T18:04:12.174 INFO:teuthology.orchestra.run.smithi110.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 10 MB/s | 5.4 MB 00:00 2024-04-23T18:04:12.233 INFO:teuthology.orchestra.run.smithi110.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 776 kB/s | 45 kB 00:00 2024-04-23T18:04:12.291 INFO:teuthology.orchestra.run.smithi110.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 2.3 MB/s | 135 kB 00:00 2024-04-23T18:04:12.358 INFO:teuthology.orchestra.run.smithi110.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.4 MB/s | 161 kB 00:00 2024-04-23T18:04:12.433 INFO:teuthology.orchestra.run.smithi110.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 4.2 MB/s | 320 kB 00:00 2024-04-23T18:04:12.517 INFO:teuthology.orchestra.run.smithi110.stdout:(23/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x8 1.5 MB/s | 4.7 MB 00:03 2024-04-23T18:04:12.542 INFO:teuthology.orchestra.run.smithi110.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.7 MB/s | 298 kB 00:00 2024-04-23T18:04:12.609 INFO:teuthology.orchestra.run.smithi110.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 85 kB 00:00 2024-04-23T18:04:12.634 INFO:teuthology.orchestra.run.smithi110.stdout:(26/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 849 kB/s | 99 kB 00:00 2024-04-23T18:04:12.651 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-23T18:04:12.818 INFO:teuthology.orchestra.run.smithi110.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 929 kB/s | 170 kB 00:00 2024-04-23T18:04:12.944 INFO:teuthology.orchestra.run.smithi110.stdout:(28/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 9.2 MB/s | 3.1 MB 00:00 2024-04-23T18:04:12.960 INFO:teuthology.orchestra.run.smithi110.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 241 kB/s | 34 kB 00:00 2024-04-23T18:04:13.002 INFO:teuthology.orchestra.run.smithi110.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 2.4 MB/s | 142 kB 00:00 2024-04-23T18:04:13.870 INFO:teuthology.orchestra.run.smithi110.stdout:(31/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 17 MB/s | 49 MB 00:02 2024-04-23T18:04:13.979 INFO:teuthology.orchestra.run.smithi110.stdout:(32/152): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-04-23T18:04:14.004 INFO:teuthology.orchestra.run.smithi110.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.8 MB/s | 249 kB 00:00 2024-04-23T18:04:14.029 INFO:teuthology.orchestra.run.smithi110.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 985 kB/s | 49 kB 00:00 2024-04-23T18:04:14.054 INFO:teuthology.orchestra.run.smithi110.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 328 kB/s | 16 kB 00:00 2024-04-23T18:04:14.079 INFO:teuthology.orchestra.run.smithi110.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 5.3 MB/s | 273 kB 00:00 2024-04-23T18:04:14.122 INFO:teuthology.orchestra.run.smithi110.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 439 kB/s | 18 kB 00:00 2024-04-23T18:04:14.155 INFO:teuthology.orchestra.run.smithi110.stdout:(38/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 7.4 MB/s | 762 kB 00:00 2024-04-23T18:04:14.289 INFO:teuthology.orchestra.run.smithi110.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-04-23T18:04:14.373 INFO:teuthology.orchestra.run.smithi110.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-23T18:04:14.398 INFO:teuthology.orchestra.run.smithi110.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-04-23T18:04:14.423 INFO:teuthology.orchestra.run.smithi110.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-23T18:04:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-23T18:04:14.516 INFO:teuthology.orchestra.run.smithi110.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.2 MB 00:00 2024-04-23T18:04:14.541 INFO:teuthology.orchestra.run.smithi110.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-23T18:04:14.566 INFO:teuthology.orchestra.run.smithi110.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-23T18:04:14.591 INFO:teuthology.orchestra.run.smithi110.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-23T18:04:14.800 INFO:teuthology.orchestra.run.smithi110.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 689 kB 00:00 2024-04-23T18:04:14.853 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:14.853 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:14.864 INFO:teuthology.orchestra.run.smithi026.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:04:14.865 INFO:teuthology.orchestra.run.smithi026.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:04:14.866 INFO:teuthology.orchestra.run.smithi026.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:04:14.867 INFO:teuthology.orchestra.run.smithi110.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 943 kB 00:00 2024-04-23T18:04:14.867 INFO:teuthology.orchestra.run.smithi026.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:04:14.869 INFO:teuthology.orchestra.run.smithi026.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:04:14.871 INFO:teuthology.orchestra.run.smithi026.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:04:14.872 INFO:teuthology.orchestra.run.smithi026.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:04:14.926 INFO:teuthology.orchestra.run.smithi110.stdout:(50/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 1.8 MB/s | 3.5 MB 00:01 2024-04-23T18:04:14.936 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-23T18:04:14.946 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-23T18:04:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-23T18:04:14.946 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-23T18:04:14.946 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-23T18:04:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:04:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:04:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:04:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:04:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:04:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:04:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:04:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:04:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:04:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:04:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:04:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:04:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout:Install 148 Packages 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 4 Packages 2024-04-23T18:04:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:04:14.961 INFO:teuthology.orchestra.run.smithi110.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 710 kB/s | 107 kB 00:00 2024-04-23T18:04:14.961 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 222 M 2024-04-23T18:04:14.961 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-23T18:04:14.977 INFO:teuthology.orchestra.run.smithi110.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-04-23T18:04:15.035 INFO:teuthology.orchestra.run.smithi110.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 562 kB/s | 33 kB 00:00 2024-04-23T18:04:15.085 INFO:teuthology.orchestra.run.smithi110.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-23T18:04:15.135 INFO:teuthology.orchestra.run.smithi110.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-23T18:04:15.161 INFO:teuthology.orchestra.run.smithi110.stdout:(56/152): flex-2.6.4-9.el9.x86_64.rpm 1.3 MB/s | 313 kB 00:00 2024-04-23T18:04:15.194 INFO:teuthology.orchestra.run.smithi110.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-23T18:04:15.219 INFO:teuthology.orchestra.run.smithi110.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-23T18:04:15.328 INFO:teuthology.orchestra.run.smithi110.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-04-23T18:04:15.353 INFO:teuthology.orchestra.run.smithi110.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 582 kB 00:00 2024-04-23T18:04:15.404 INFO:teuthology.orchestra.run.smithi110.stdout:(61/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.0 MB/s | 3.0 MB 00:00 2024-04-23T18:04:15.421 INFO:teuthology.orchestra.run.smithi110.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-04-23T18:04:15.437 INFO:teuthology.orchestra.run.smithi110.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-23T18:04:15.463 INFO:teuthology.orchestra.run.smithi110.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-23T18:04:15.488 INFO:teuthology.orchestra.run.smithi110.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 188 kB 00:00 2024-04-23T18:04:15.513 INFO:teuthology.orchestra.run.smithi110.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 466 kB/s | 35 kB 00:00 2024-04-23T18:04:15.620 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-23T18:04:15.731 INFO:teuthology.orchestra.run.smithi110.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-23T18:04:15.820 INFO:teuthology.orchestra.run.smithi026.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 36 kB/s | 6.6 kB 00:00 2024-04-23T18:04:15.864 INFO:teuthology.orchestra.run.smithi110.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-23T18:04:15.923 INFO:teuthology.orchestra.run.smithi110.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-23T18:04:15.990 INFO:teuthology.orchestra.run.smithi110.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 9.0 MB/s | 4.7 MB 00:00 2024-04-23T18:04:16.011 INFO:teuthology.orchestra.run.smithi026.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 4.5 MB/s | 890 kB 00:00 2024-04-23T18:04:16.062 INFO:teuthology.orchestra.run.smithi026.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-23T18:04:16.082 INFO:teuthology.orchestra.run.smithi110.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-04-23T18:04:16.140 INFO:teuthology.orchestra.run.smithi110.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-23T18:04:16.183 INFO:teuthology.orchestra.run.smithi110.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-23T18:04:16.241 INFO:teuthology.orchestra.run.smithi110.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-23T18:04:16.287 INFO:teuthology.orchestra.run.smithi026.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 10 MB/s | 2.4 MB 00:00 2024-04-23T18:04:16.308 INFO:teuthology.orchestra.run.smithi110.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-04-23T18:04:16.366 INFO:teuthology.orchestra.run.smithi110.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 441 kB/s | 26 kB 00:00 2024-04-23T18:04:16.417 INFO:teuthology.orchestra.run.smithi110.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-23T18:04:16.470 INFO:teuthology.orchestra.run.smithi026.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-23T18:04:16.475 INFO:teuthology.orchestra.run.smithi110.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-23T18:04:16.534 INFO:teuthology.orchestra.run.smithi110.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 596 kB/s | 35 kB 00:00 2024-04-23T18:04:16.618 INFO:teuthology.orchestra.run.smithi110.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 8.3 MB/s | 5.8 MB 00:00 2024-04-23T18:04:16.676 INFO:teuthology.orchestra.run.smithi110.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-23T18:04:16.752 INFO:teuthology.orchestra.run.smithi110.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-23T18:04:16.810 INFO:teuthology.orchestra.run.smithi110.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T18:04:16.854 INFO:teuthology.orchestra.run.smithi026.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 4.4 MB/s | 5.4 MB 00:01 2024-04-23T18:04:16.869 INFO:teuthology.orchestra.run.smithi110.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 725 kB/s | 42 kB 00:00 2024-04-23T18:04:16.929 INFO:teuthology.orchestra.run.smithi026.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 10 MB/s | 4.7 MB 00:00 2024-04-23T18:04:16.944 INFO:teuthology.orchestra.run.smithi110.stdout:(85/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 6.0 MB 00:00 2024-04-23T18:04:16.970 INFO:teuthology.orchestra.run.smithi110.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 269 kB 00:00 2024-04-23T18:04:17.003 INFO:teuthology.orchestra.run.smithi110.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-23T18:04:17.028 INFO:teuthology.orchestra.run.smithi110.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-23T18:04:17.053 INFO:teuthology.orchestra.run.smithi110.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-23T18:04:17.087 INFO:teuthology.orchestra.run.smithi110.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 917 kB/s | 54 kB 00:00 2024-04-23T18:04:17.162 INFO:teuthology.orchestra.run.smithi110.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.1 MB/s | 5.1 MB 00:00 2024-04-23T18:04:17.188 INFO:teuthology.orchestra.run.smithi110.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-23T18:04:17.230 INFO:teuthology.orchestra.run.smithi110.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 661 kB/s | 44 kB 00:00 2024-04-23T18:04:17.255 INFO:teuthology.orchestra.run.smithi110.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-23T18:04:17.289 INFO:teuthology.orchestra.run.smithi110.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 612 kB/s | 36 kB 00:00 2024-04-23T18:04:17.322 INFO:teuthology.orchestra.run.smithi110.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 705 kB/s | 47 kB 00:00 2024-04-23T18:04:17.348 INFO:teuthology.orchestra.run.smithi110.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-23T18:04:17.410 INFO:teuthology.orchestra.run.smithi110.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 664 kB/s | 58 kB 00:00 2024-04-23T18:04:17.440 INFO:teuthology.orchestra.run.smithi110.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 94 kB 00:00 2024-04-23T18:04:17.465 INFO:teuthology.orchestra.run.smithi110.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T18:04:17.499 INFO:teuthology.orchestra.run.smithi110.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 764 kB/s | 45 kB 00:00 2024-04-23T18:04:17.549 INFO:teuthology.orchestra.run.smithi110.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-04-23T18:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-04-23T18:04:17.867 INFO:teuthology.orchestra.run.smithi110.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-23T18:04:17.873 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-23T18:04:18.017 INFO:teuthology.orchestra.run.smithi110.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-04-23T18:04:18.051 INFO:teuthology.orchestra.run.smithi110.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 881 kB/s | 309 kB 00:00 2024-04-23T18:04:18.101 INFO:teuthology.orchestra.run.smithi110.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-04-23T18:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-04-23T18:04:18.260 INFO:teuthology.orchestra.run.smithi110.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-23T18:04:18.285 INFO:teuthology.orchestra.run.smithi110.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-04-23T18:04:18.310 INFO:teuthology.orchestra.run.smithi110.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-23T18:04:18.313 INFO:teuthology.orchestra.run.smithi026.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 7.9 MB/s | 21 MB 00:02 2024-04-23T18:04:18.430 INFO:teuthology.orchestra.run.smithi026.stdout:(9/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 7.1 MB/s | 11 MB 00:01 2024-04-23T18:04:18.452 INFO:teuthology.orchestra.run.smithi110.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-23T18:04:18.455 INFO:teuthology.orchestra.run.smithi026.stdout:(10/152): ceph-selinux-19.0.0-3156.gd69f13d8.el 177 kB/s | 25 kB 00:00 2024-04-23T18:04:18.527 INFO:teuthology.orchestra.run.smithi110.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 581 kB/s | 43 kB 00:00 2024-04-23T18:04:18.553 INFO:teuthology.orchestra.run.smithi110.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.2 MB/s | 548 kB 00:00 2024-04-23T18:04:18.564 INFO:teuthology.orchestra.run.smithi026.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.0 MB/s | 225 kB 00:00 2024-04-23T18:04:18.612 INFO:teuthology.orchestra.run.smithi110.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 384 kB/s | 32 kB 00:00 2024-04-23T18:04:18.637 INFO:teuthology.orchestra.run.smithi110.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-23T18:04:18.655 INFO:teuthology.orchestra.run.smithi026.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-23T18:04:18.679 INFO:teuthology.orchestra.run.smithi110.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-23T18:04:18.704 INFO:teuthology.orchestra.run.smithi110.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-23T18:04:18.755 INFO:teuthology.orchestra.run.smithi110.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-23T18:04:18.771 INFO:teuthology.orchestra.run.smithi110.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 526 kB/s | 35 kB 00:00 2024-04-23T18:04:18.822 INFO:teuthology.orchestra.run.smithi110.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-23T18:04:18.831 INFO:teuthology.orchestra.run.smithi026.stdout:(13/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 8.5 MB/s | 17 MB 00:01 2024-04-23T18:04:18.856 INFO:teuthology.orchestra.run.smithi026.stdout:(14/152): libcephfs-devel-19.0.0-3156.gd69f13d8 158 kB/s | 32 kB 00:00 2024-04-23T18:04:18.957 INFO:teuthology.orchestra.run.smithi026.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 163 kB 00:00 2024-04-23T18:04:18.989 INFO:teuthology.orchestra.run.smithi110.stdout:(122/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-04-23T18:04:19.023 INFO:teuthology.orchestra.run.smithi110.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 720 kB/s | 144 kB 00:00 2024-04-23T18:04:19.031 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-23T18:04:19.040 INFO:teuthology.orchestra.run.smithi026.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-23T18:04:19.065 INFO:teuthology.orchestra.run.smithi026.stdout:(17/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 3.1 MB/s | 747 kB 00:00 2024-04-23T18:04:19.106 INFO:teuthology.orchestra.run.smithi110.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-04-23T18:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 216 kB/s | 18 kB 00:00 2024-04-23T18:04:19.157 INFO:teuthology.orchestra.run.smithi110.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 471 kB/s | 23 kB 00:00 2024-04-23T18:04:19.157 INFO:teuthology.orchestra.run.smithi026.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 4.2 MB/s | 500 kB 00:00 2024-04-23T18:04:19.182 INFO:teuthology.orchestra.run.smithi110.stdout:(127/152): python3-jaraco-8.2.1-3.el9.noarch.rp 55 kB/s | 11 kB 00:00 2024-04-23T18:04:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-23T18:04:19.233 INFO:teuthology.orchestra.run.smithi110.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-23T18:04:19.233 INFO:teuthology.orchestra.run.smithi026.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 601 kB/s | 45 kB 00:00 2024-04-23T18:04:19.258 INFO:teuthology.orchestra.run.smithi110.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 924 kB/s | 46 kB 00:00 2024-04-23T18:04:19.300 INFO:teuthology.orchestra.run.smithi110.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-23T18:04:19.309 INFO:teuthology.orchestra.run.smithi026.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.8 MB/s | 135 kB 00:00 2024-04-23T18:04:19.325 INFO:teuthology.orchestra.run.smithi110.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-04-23T18:04:19.350 INFO:teuthology.orchestra.run.smithi110.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-04-23T18:04:19.376 INFO:teuthology.orchestra.run.smithi110.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-23T18:04:19.428 INFO:teuthology.orchestra.run.smithi026.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.9 MB/s | 161 kB 00:00 2024-04-23T18:04:19.429 INFO:teuthology.orchestra.run.smithi110.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-23T18:04:19.434 INFO:teuthology.orchestra.run.smithi110.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-04-23T18:04:19.460 INFO:teuthology.orchestra.run.smithi110.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 825 kB/s | 90 kB 00:00 2024-04-23T18:04:19.485 INFO:teuthology.orchestra.run.smithi110.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-23T18:04:19.492 INFO:teuthology.orchestra.run.smithi026.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.1 MB/s | 320 kB 00:00 2024-04-23T18:04:19.510 INFO:teuthology.orchestra.run.smithi110.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-23T18:04:19.536 INFO:teuthology.orchestra.run.smithi110.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-04-23T18:04:19.561 INFO:teuthology.orchestra.run.smithi110.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-23T18:04:19.584 INFO:teuthology.orchestra.run.smithi026.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 298 kB 00:00 2024-04-23T18:04:19.586 INFO:teuthology.orchestra.run.smithi110.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-23T18:04:19.612 INFO:teuthology.orchestra.run.smithi110.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-23T18:04:19.645 INFO:teuthology.orchestra.run.smithi110.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-23T18:04:19.659 INFO:teuthology.orchestra.run.smithi026.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 99 kB 00:00 2024-04-23T18:04:19.670 INFO:teuthology.orchestra.run.smithi110.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-04-23T18:04:19.696 INFO:teuthology.orchestra.run.smithi110.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-23T18:04:19.721 INFO:teuthology.orchestra.run.smithi110.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-23T18:04:19.726 INFO:teuthology.orchestra.run.smithi026.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 85 kB 00:00 2024-04-23T18:04:19.771 INFO:teuthology.orchestra.run.smithi110.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-23T18:04:19.951 INFO:teuthology.orchestra.run.smithi026.stdout:(26/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 6.1 MB/s | 5.4 MB 00:00 2024-04-23T18:04:19.980 INFO:teuthology.orchestra.run.smithi110.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-04-23T18:04:20.068 INFO:teuthology.orchestra.run.smithi026.stdout:(27/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 9.0 MB/s | 3.1 MB 00:00 2024-04-23T18:04:20.089 INFO:teuthology.orchestra.run.smithi110.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2024-04-23T18:04:20.102 INFO:teuthology.orchestra.run.smithi026.stdout:(28/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.1 MB/s | 170 kB 00:00 2024-04-23T18:04:20.127 INFO:teuthology.orchestra.run.smithi026.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 583 kB/s | 34 kB 00:00 2024-04-23T18:04:20.194 INFO:teuthology.orchestra.run.smithi026.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.5 MB/s | 142 kB 00:00 2024-04-23T18:04:20.231 INFO:teuthology.orchestra.run.smithi110.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 6.2 MB/s | 3.1 MB 00:00 2024-04-23T18:04:20.486 INFO:teuthology.orchestra.run.smithi026.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 9.9 MB/s | 3.5 MB 00:00 2024-04-23T18:04:20.578 INFO:teuthology.orchestra.run.smithi026.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-23T18:04:20.662 INFO:teuthology.orchestra.run.smithi026.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 592 kB/s | 49 kB 00:00 2024-04-23T18:04:20.737 INFO:teuthology.orchestra.run.smithi026.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 219 kB/s | 16 kB 00:00 2024-04-23T18:04:20.821 INFO:teuthology.orchestra.run.smithi026.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 273 kB 00:00 2024-04-23T18:04:20.921 INFO:teuthology.orchestra.run.smithi026.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 7.5 MB/s | 762 kB 00:00 2024-04-23T18:04:20.996 INFO:teuthology.orchestra.run.smithi026.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 243 kB/s | 18 kB 00:00 2024-04-23T18:04:21.205 INFO:teuthology.orchestra.run.smithi026.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-04-23T18:04:21.230 INFO:teuthology.orchestra.run.smithi026.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 80 kB 00:00 2024-04-23T18:04:21.347 INFO:teuthology.orchestra.run.smithi026.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-23T18:04:21.430 INFO:teuthology.orchestra.run.smithi026.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-23T18:04:21.481 INFO:teuthology.orchestra.run.smithi110.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 1.9 MB/s | 3.4 MB 00:01 2024-04-23T18:04:21.486 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:04:21.487 INFO:teuthology.orchestra.run.smithi110.stdout:Total 16 MB/s | 222 MB 00:13 2024-04-23T18:04:21.497 INFO:teuthology.orchestra.run.smithi026.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-04-23T18:04:21.573 INFO:teuthology.orchestra.run.smithi026.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-23T18:04:21.648 INFO:teuthology.orchestra.run.smithi026.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-23T18:04:21.715 INFO:teuthology.orchestra.run.smithi026.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 4.3 MB/s | 1.2 MB 00:00 2024-04-23T18:04:21.740 INFO:teuthology.orchestra.run.smithi026.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-04-23T18:04:21.774 INFO:teuthology.orchestra.run.smithi026.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-23T18:04:22.483 INFO:teuthology.orchestra.run.smithi026.stdout:(48/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 12 MB/s | 49 MB 00:04 2024-04-23T18:04:22.544 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-04-23T18:04:22.688 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-04-23T18:04:22.688 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-04-23T18:04:23.233 INFO:teuthology.orchestra.run.smithi026.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 143 kB/s | 107 kB 00:00 2024-04-23T18:04:23.633 INFO:teuthology.orchestra.run.smithi026.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 364 kB/s | 689 kB 00:01 2024-04-23T18:04:23.858 INFO:teuthology.orchestra.run.smithi026.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 34 kB/s | 21 kB 00:00 2024-04-23T18:04:23.958 INFO:teuthology.orchestra.run.smithi026.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 432 kB/s | 943 kB 00:02 2024-04-23T18:04:24.182 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-23T18:04:24.461 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-04-23T18:04:24.462 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-04-23T18:04:24.492 INFO:teuthology.orchestra.run.smithi026.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-04-23T18:04:24.634 INFO:teuthology.orchestra.run.smithi026.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 314 kB/s | 313 kB 00:00 2024-04-23T18:04:25.018 INFO:teuthology.orchestra.run.smithi026.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-04-23T18:04:25.176 INFO:teuthology.orchestra.run.smithi026.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-04-23T18:04:25.835 INFO:teuthology.orchestra.run.smithi026.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 69 kB/s | 45 kB 00:00 2024-04-23T18:04:25.860 INFO:teuthology.orchestra.run.smithi026.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 190 kB/s | 160 kB 00:00 2024-04-23T18:04:26.308 INFO:teuthology.orchestra.run.smithi131.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:26.309 INFO:teuthology.orchestra.run.smithi131.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:04:26.320 INFO:teuthology.orchestra.run.smithi131.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:04:26.320 INFO:teuthology.orchestra.run.smithi131.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:04:26.321 INFO:teuthology.orchestra.run.smithi131.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:04:26.322 INFO:teuthology.orchestra.run.smithi131.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:04:26.324 INFO:teuthology.orchestra.run.smithi131.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:04:26.326 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:04:26.327 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:04:26.387 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:04:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:04:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:04:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrading: 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-04-23T18:04:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:04:26.411 INFO:teuthology.orchestra.run.smithi131.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:04:26.412 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:04:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:04:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:04:26.415 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:04:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:04:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout:Installing weak dependencies: 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:04:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:======================================================================================= 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:Install 148 Packages 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrade 4 Packages 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 222 M 2024-04-23T18:04:26.419 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-04-23T18:04:27.169 INFO:teuthology.orchestra.run.smithi026.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 445 kB/s | 582 kB 00:01 2024-04-23T18:04:27.192 INFO:teuthology.orchestra.run.smithi131.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-23T18:04:27.219 INFO:teuthology.orchestra.run.smithi026.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 479 kB/s | 662 kB 00:01 2024-04-23T18:04:27.528 INFO:teuthology.orchestra.run.smithi131.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 2.6 MB/s | 890 kB 00:00 2024-04-23T18:04:27.592 INFO:teuthology.orchestra.run.smithi131.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.2 MB/s | 145 kB 00:00 2024-04-23T18:04:27.761 INFO:teuthology.orchestra.run.smithi026.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-04-23T18:04:27.869 INFO:teuthology.orchestra.run.smithi026.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-04-23T18:04:27.934 INFO:teuthology.orchestra.run.smithi131.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 6.9 MB/s | 2.4 MB 00:00 2024-04-23T18:04:28.143 INFO:teuthology.orchestra.run.smithi131.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 7.2 MB/s | 1.5 MB 00:00 2024-04-23T18:04:28.370 INFO:teuthology.orchestra.run.smithi026.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 694 kB/s | 3.0 MB 00:04 2024-04-23T18:04:28.718 INFO:teuthology.orchestra.run.smithi131.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 3.2 MB/s | 5.4 MB 00:01 2024-04-23T18:04:28.728 INFO:teuthology.orchestra.run.smithi026.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 219 kB/s | 188 kB 00:00 2024-04-23T18:04:28.753 INFO:teuthology.orchestra.run.smithi026.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 295 kB/s | 292 kB 00:00 2024-04-23T18:04:28.785 INFO:teuthology.orchestra.run.smithi131.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 7.3 MB/s | 4.7 MB 00:00 2024-04-23T18:04:29.045 INFO:teuthology.orchestra.run.smithi026.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 52 kB/s | 35 kB 00:00 2024-04-23T18:04:29.102 INFO:teuthology.orchestra.run.smithi131.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 10 MB/s | 21 MB 00:02 2024-04-23T18:04:29.144 INFO:teuthology.orchestra.run.smithi131.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 609 kB/s | 25 kB 00:00 2024-04-23T18:04:29.944 INFO:teuthology.orchestra.run.smithi131.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 9.1 MB/s | 11 MB 00:01 2024-04-23T18:04:30.019 INFO:teuthology.orchestra.run.smithi131.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.9 MB/s | 225 kB 00:00 2024-04-23T18:04:30.086 INFO:teuthology.orchestra.run.smithi131.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 407 kB/s | 27 kB 00:00 2024-04-23T18:04:30.154 INFO:teuthology.orchestra.run.smithi131.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 472 kB/s | 32 kB 00:00 2024-04-23T18:04:30.295 INFO:teuthology.orchestra.run.smithi131.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 5.2 MB/s | 747 kB 00:00 2024-04-23T18:04:30.371 INFO:teuthology.orchestra.run.smithi131.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.1 MB/s | 163 kB 00:00 2024-04-23T18:04:30.386 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-04-23T18:04:30.438 INFO:teuthology.orchestra.run.smithi131.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.9 MB/s | 126 kB 00:00 2024-04-23T18:04:30.468 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:04:30.506 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:04:30.530 INFO:teuthology.orchestra.run.smithi131.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 5.3 MB/s | 500 kB 00:00 2024-04-23T18:04:30.643 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:04:30.744 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:04:30.769 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:04:30.811 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:04:31.139 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:04:31.205 INFO:teuthology.orchestra.run.smithi131.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 7.9 MB/s | 5.4 MB 00:00 2024-04-23T18:04:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:04:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 540 kB/s | 45 kB 00:00 2024-04-23T18:04:31.364 INFO:teuthology.orchestra.run.smithi131.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.8 MB/s | 135 kB 00:00 2024-04-23T18:04:31.384 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:04:31.413 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:04:31.439 INFO:teuthology.orchestra.run.smithi131.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.1 MB/s | 161 kB 00:00 2024-04-23T18:04:31.505 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:04:31.523 INFO:teuthology.orchestra.run.smithi131.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.8 MB/s | 320 kB 00:00 2024-04-23T18:04:31.534 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:04:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:04:31.724 INFO:teuthology.orchestra.run.smithi131.stdout:(23/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 5.6 MB/s | 17 MB 00:03 2024-04-23T18:04:31.737 INFO:teuthology.orchestra.run.smithi026.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 604 kB/s | 1.6 MB 00:02 2024-04-23T18:04:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:04:31.749 INFO:teuthology.orchestra.run.smithi131.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 298 kB 00:00 2024-04-23T18:04:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:04:31.791 INFO:teuthology.orchestra.run.smithi131.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 2.0 MB/s | 85 kB 00:00 2024-04-23T18:04:31.816 INFO:teuthology.orchestra.run.smithi131.stdout:(26/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 99 kB 00:00 2024-04-23T18:04:31.854 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:04:31.877 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:04:31.916 INFO:teuthology.orchestra.run.smithi131.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 170 kB 00:00 2024-04-23T18:04:31.931 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:04:31.961 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:04:31.986 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:04:32.178 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:04:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:04:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:04:32.383 INFO:teuthology.orchestra.run.smithi131.stdout:(28/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 15 MB/s | 49 MB 00:03 2024-04-23T18:04:32.409 INFO:teuthology.orchestra.run.smithi131.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 69 kB/s | 34 kB 00:00 2024-04-23T18:04:32.423 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:04:32.434 INFO:teuthology.orchestra.run.smithi131.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 2.8 MB/s | 142 kB 00:00 2024-04-23T18:04:32.469 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:04:32.484 INFO:teuthology.orchestra.run.smithi131.stdout:(31/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 4.5 MB/s | 3.1 MB 00:00 2024-04-23T18:04:32.521 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:04:32.576 INFO:teuthology.orchestra.run.smithi131.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-23T18:04:32.594 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:04:32.637 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:04:32.643 INFO:teuthology.orchestra.run.smithi131.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 740 kB/s | 49 kB 00:00 2024-04-23T18:04:32.689 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:04:32.718 INFO:teuthology.orchestra.run.smithi131.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 219 kB/s | 16 kB 00:00 2024-04-23T18:04:32.729 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:04:32.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:04:32.802 INFO:teuthology.orchestra.run.smithi131.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 273 kB 00:00 2024-04-23T18:04:32.827 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:04:32.889 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:04:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:04:32.944 INFO:teuthology.orchestra.run.smithi131.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-04-23T18:04:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:04:33.003 INFO:teuthology.orchestra.run.smithi131.stdout:(37/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 6.0 MB/s | 3.5 MB 00:00 2024-04-23T18:04:33.028 INFO:teuthology.orchestra.run.smithi131.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 219 kB/s | 18 kB 00:00 2024-04-23T18:04:33.039 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:04:33.053 INFO:teuthology.orchestra.run.smithi131.stdout:(39/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 3.0 MB/s | 762 kB 00:00 2024-04-23T18:04:33.073 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:04:33.117 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:04:33.145 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:04:33.178 INFO:teuthology.orchestra.run.smithi131.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 458 kB/s | 80 kB 00:00 2024-04-23T18:04:33.214 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:04:33.262 INFO:teuthology.orchestra.run.smithi131.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-04-23T18:04:33.268 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:04:33.295 INFO:teuthology.orchestra.run.smithi131.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-23T18:04:33.329 INFO:teuthology.orchestra.run.smithi131.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-23T18:04:33.342 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:04:33.355 INFO:teuthology.orchestra.run.smithi131.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-23T18:04:33.388 INFO:teuthology.orchestra.run.smithi131.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T18:04:33.414 INFO:teuthology.orchestra.run.smithi131.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-23T18:04:33.447 INFO:teuthology.orchestra.run.smithi131.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-23T18:04:33.470 INFO:teuthology.orchestra.run.smithi026.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 526 kB/s | 911 kB 00:01 2024-04-23T18:04:33.481 INFO:teuthology.orchestra.run.smithi131.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-04-23T18:04:33.483 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:04:33.537 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:04:33.598 INFO:teuthology.orchestra.run.smithi131.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-04-23T18:04:33.623 INFO:teuthology.orchestra.run.smithi131.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 754 kB/s | 107 kB 00:00 2024-04-23T18:04:33.649 INFO:teuthology.orchestra.run.smithi131.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 427 kB/s | 21 kB 00:00 2024-04-23T18:04:33.682 INFO:teuthology.orchestra.run.smithi131.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-04-23T18:04:33.699 INFO:teuthology.orchestra.run.smithi131.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-04-23T18:04:33.725 INFO:teuthology.orchestra.run.smithi131.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-04-23T18:04:33.741 INFO:teuthology.orchestra.run.smithi131.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 435 kB/s | 18 kB 00:00 2024-04-23T18:04:33.758 INFO:teuthology.orchestra.run.smithi131.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 331 kB/s | 11 kB 00:00 2024-04-23T18:04:33.792 INFO:teuthology.orchestra.run.smithi131.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-23T18:04:33.817 INFO:teuthology.orchestra.run.smithi131.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-23T18:04:33.884 INFO:teuthology.orchestra.run.smithi131.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-04-23T18:04:33.934 INFO:teuthology.orchestra.run.smithi131.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 511 kB/s | 25 kB 00:00 2024-04-23T18:04:34.002 INFO:teuthology.orchestra.run.smithi131.stdout:(61/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.4 MB/s | 3.0 MB 00:00 2024-04-23T18:04:34.027 INFO:teuthology.orchestra.run.smithi131.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 515 kB/s | 47 kB 00:00 2024-04-23T18:04:34.052 INFO:teuthology.orchestra.run.smithi131.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-23T18:04:34.065 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:04:34.086 INFO:teuthology.orchestra.run.smithi131.stdout:(64/152): libtool-2.4.6-45.el9.x86_64.rpm 2.1 MB/s | 582 kB 00:00 2024-04-23T18:04:34.111 INFO:teuthology.orchestra.run.smithi131.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-23T18:04:34.145 INFO:teuthology.orchestra.run.smithi131.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-23T18:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:04:34.245 INFO:teuthology.orchestra.run.smithi026.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 172 kB/s | 133 kB 00:00 2024-04-23T18:04:34.251 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:04:34.293 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:04:34.312 INFO:teuthology.orchestra.run.smithi131.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-23T18:04:34.420 INFO:teuthology.orchestra.run.smithi131.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-23T18:04:34.471 INFO:teuthology.orchestra.run.smithi131.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-23T18:04:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:04:34.546 INFO:teuthology.orchestra.run.smithi131.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-23T18:04:34.546 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:04:34.601 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:04:34.629 INFO:teuthology.orchestra.run.smithi131.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.7 MB/s | 4.7 MB 00:00 2024-04-23T18:04:34.680 INFO:teuthology.orchestra.run.smithi131.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-23T18:04:34.682 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:04:34.721 INFO:teuthology.orchestra.run.smithi131.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-04-23T18:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:04:34.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:04:34.781 INFO:teuthology.orchestra.run.smithi131.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-23T18:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:04:34.831 INFO:teuthology.orchestra.run.smithi131.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-23T18:04:34.841 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:04:34.881 INFO:teuthology.orchestra.run.smithi131.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 515 kB/s | 26 kB 00:00 2024-04-23T18:04:34.884 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:04:34.923 INFO:teuthology.orchestra.run.smithi131.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 458 kB/s | 19 kB 00:00 2024-04-23T18:04:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:04:34.974 INFO:teuthology.orchestra.run.smithi131.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-23T18:04:35.024 INFO:teuthology.orchestra.run.smithi131.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-23T18:04:35.054 INFO:teuthology.orchestra.run.smithi026.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 719 kB/s | 4.4 MB 00:06 2024-04-23T18:04:35.063 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:04:35.116 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:04:35.142 INFO:teuthology.orchestra.run.smithi131.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 8.6 MB/s | 5.8 MB 00:00 2024-04-23T18:04:35.146 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:04:35.188 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:04:35.191 INFO:teuthology.orchestra.run.smithi131.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-23T18:04:35.241 INFO:teuthology.orchestra.run.smithi131.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-23T18:04:35.289 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:04:35.291 INFO:teuthology.orchestra.run.smithi131.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-23T18:04:35.329 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:04:35.342 INFO:teuthology.orchestra.run.smithi131.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-04-23T18:04:35.369 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:04:35.392 INFO:teuthology.orchestra.run.smithi131.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-23T18:04:35.395 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:04:35.420 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:04:35.451 INFO:teuthology.orchestra.run.smithi131.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-23T18:04:35.494 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:04:35.534 INFO:teuthology.orchestra.run.smithi131.stdout:(87/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.0 MB/s | 6.0 MB 00:00 2024-04-23T18:04:35.601 INFO:teuthology.orchestra.run.smithi131.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.8 MB/s | 5.1 MB 00:00 2024-04-23T18:04:35.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:04:35.627 INFO:teuthology.orchestra.run.smithi131.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-04-23T18:04:35.652 INFO:teuthology.orchestra.run.smithi131.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-04-23T18:04:35.652 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:04:35.654 INFO:teuthology.orchestra.run.smithi026.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 701 kB/s | 4.7 MB 00:06 2024-04-23T18:04:35.680 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:04:35.685 INFO:teuthology.orchestra.run.smithi131.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 917 kB/s | 54 kB 00:00 2024-04-23T18:04:35.714 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:04:35.744 INFO:teuthology.orchestra.run.smithi131.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-04-23T18:04:35.754 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:04:35.794 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:04:35.811 INFO:teuthology.orchestra.run.smithi131.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-23T18:04:35.836 INFO:teuthology.orchestra.run.smithi131.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 293 kB/s | 44 kB 00:00 2024-04-23T18:04:35.887 INFO:teuthology.orchestra.run.smithi131.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-23T18:04:35.912 INFO:teuthology.orchestra.run.smithi131.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 357 kB/s | 36 kB 00:00 2024-04-23T18:04:35.987 INFO:teuthology.orchestra.run.smithi131.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 305 kB 00:00 2024-04-23T18:04:35.991 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:04:36.013 INFO:teuthology.orchestra.run.smithi131.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 577 kB/s | 58 kB 00:00 2024-04-23T18:04:36.038 INFO:teuthology.orchestra.run.smithi131.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-23T18:04:36.063 INFO:teuthology.orchestra.run.smithi131.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-23T18:04:36.089 INFO:teuthology.orchestra.run.smithi131.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 886 kB/s | 45 kB 00:00 2024-04-23T18:04:36.143 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:04:36.176 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:04:36.218 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:04:36.252 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:04:36.343 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:04:36.354 INFO:teuthology.orchestra.run.smithi026.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 125 kB/s | 87 kB 00:00 2024-04-23T18:04:36.356 INFO:teuthology.orchestra.run.smithi131.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-04-23T18:04:36.390 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:04:36.421 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:04:36.469 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:04:36.575 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:04:36.631 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:04:36.681 INFO:teuthology.orchestra.run.smithi131.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-23T18:04:36.740 INFO:teuthology.orchestra.run.smithi131.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 5.2 MB/s | 309 kB 00:00 2024-04-23T18:04:36.798 INFO:teuthology.orchestra.run.smithi131.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-04-23T18:04:36.823 INFO:teuthology.orchestra.run.smithi131.stdout:(106/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-04-23T18:04:36.954 INFO:teuthology.orchestra.run.smithi026.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 544 kB/s | 1.0 MB 00:01 2024-04-23T18:04:37.016 INFO:teuthology.orchestra.run.smithi131.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 135 kB/s | 26 kB 00:00 2024-04-23T18:04:37.149 INFO:teuthology.orchestra.run.smithi131.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-23T18:04:37.175 INFO:teuthology.orchestra.run.smithi131.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-04-23T18:04:37.333 INFO:teuthology.orchestra.run.smithi131.stdout:(110/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.5 MB/s | 16 MB 00:01 2024-04-23T18:04:37.339 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:04:37.358 INFO:teuthology.orchestra.run.smithi131.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 322 kB/s | 67 kB 00:00 2024-04-23T18:04:37.378 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:04:37.383 INFO:teuthology.orchestra.run.smithi131.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 723 kB/s | 151 kB 00:00 2024-04-23T18:04:37.412 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:04:37.434 INFO:teuthology.orchestra.run.smithi131.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-23T18:04:37.468 INFO:teuthology.orchestra.run.smithi131.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-04-23T18:04:37.493 INFO:teuthology.orchestra.run.smithi131.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 548 kB/s | 32 kB 00:00 2024-04-23T18:04:37.510 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:04:37.527 INFO:teuthology.orchestra.run.smithi131.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-04-23T18:04:37.552 INFO:teuthology.orchestra.run.smithi131.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-23T18:04:37.578 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:04:37.635 INFO:teuthology.orchestra.run.smithi131.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-23T18:04:37.661 INFO:teuthology.orchestra.run.smithi131.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-04-23T18:04:37.673 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:04:37.703 INFO:teuthology.orchestra.run.smithi131.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 526 kB/s | 35 kB 00:00 2024-04-23T18:04:37.706 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:04:37.770 INFO:teuthology.orchestra.run.smithi131.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-04-23T18:04:37.862 INFO:teuthology.orchestra.run.smithi131.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-23T18:04:37.882 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:04:37.888 INFO:teuthology.orchestra.run.smithi026.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 262 kB/s | 244 kB 00:00 2024-04-23T18:04:37.895 INFO:teuthology.orchestra.run.smithi131.stdout:(123/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-04-23T18:04:37.912 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:04:37.937 INFO:teuthology.orchestra.run.smithi131.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-23T18:04:37.971 INFO:teuthology.orchestra.run.smithi131.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-23T18:04:37.988 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:04:37.996 INFO:teuthology.orchestra.run.smithi131.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-04-23T18:04:38.022 INFO:teuthology.orchestra.run.smithi131.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-04-23T18:04:38.047 INFO:teuthology.orchestra.run.smithi131.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 256 kB/s | 19 kB 00:00 2024-04-23T18:04:38.081 INFO:teuthology.orchestra.run.smithi131.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 234 kB/s | 20 kB 00:00 2024-04-23T18:04:38.106 INFO:teuthology.orchestra.run.smithi131.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 789 kB/s | 46 kB 00:00 2024-04-23T18:04:38.148 INFO:teuthology.orchestra.run.smithi131.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-04-23T18:04:38.173 INFO:teuthology.orchestra.run.smithi131.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 854 kB/s | 79 kB 00:00 2024-04-23T18:04:38.198 INFO:teuthology.orchestra.run.smithi131.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2024-04-23T18:04:38.224 INFO:teuthology.orchestra.run.smithi131.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-23T18:04:38.241 INFO:teuthology.orchestra.run.smithi131.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 244 kB/s | 16 kB 00:00 2024-04-23T18:04:38.266 INFO:teuthology.orchestra.run.smithi131.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-23T18:04:38.283 INFO:teuthology.orchestra.run.smithi131.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 477 kB/s | 28 kB 00:00 2024-04-23T18:04:38.300 INFO:teuthology.orchestra.run.smithi131.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 523 kB/s | 31 kB 00:00 2024-04-23T18:04:38.333 INFO:teuthology.orchestra.run.smithi131.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-23T18:04:38.359 INFO:teuthology.orchestra.run.smithi131.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-23T18:04:38.384 INFO:teuthology.orchestra.run.smithi131.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 426 kB/s | 36 kB 00:00 2024-04-23T18:04:38.409 INFO:teuthology.orchestra.run.smithi131.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-23T18:04:38.443 INFO:teuthology.orchestra.run.smithi131.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-23T18:04:38.502 INFO:teuthology.orchestra.run.smithi131.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-23T18:04:38.530 INFO:teuthology.orchestra.run.smithi131.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-04-23T18:04:38.552 INFO:teuthology.orchestra.run.smithi131.stdout:(146/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-23T18:04:38.578 INFO:teuthology.orchestra.run.smithi131.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-23T18:04:38.622 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:04:38.744 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:04:38.836 INFO:teuthology.orchestra.run.smithi131.stdout:(148/152): librados2-19.0.0-3156.gd69f13d8.el7. 12 MB/s | 3.4 MB 00:00 2024-04-23T18:04:38.838 INFO:teuthology.orchestra.run.smithi026.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 261 kB/s | 248 kB 00:00 2024-04-23T18:04:38.961 INFO:teuthology.orchestra.run.smithi131.stdout:(149/152): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-04-23T18:04:39.262 INFO:teuthology.orchestra.run.smithi131.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 1.3 MB 00:00 2024-04-23T18:04:39.555 INFO:teuthology.orchestra.run.smithi026.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 36 kB/s | 26 kB 00:00 2024-04-23T18:04:39.603 INFO:teuthology.orchestra.run.smithi131.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 3.1 MB/s | 3.1 MB 00:01 2024-04-23T18:04:39.645 INFO:teuthology.orchestra.run.smithi131.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.5 MB/s | 2.4 MB 00:00 2024-04-23T18:04:39.652 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:04:39.652 INFO:teuthology.orchestra.run.smithi131.stdout:Total 17 MB/s | 222 MB 00:13 2024-04-23T18:04:39.958 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:04:40.041 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:04:40.078 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:04:40.119 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:04:40.305 INFO:teuthology.orchestra.run.smithi026.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 26 kB/s | 19 kB 00:00 2024-04-23T18:04:40.440 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:04:40.482 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:04:40.556 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:04:40.587 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:04:40.623 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:04:40.766 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-04-23T18:04:40.909 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-04-23T18:04:40.910 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-04-23T18:04:41.102 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:04:41.127 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:04:41.198 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:04:41.237 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:04:41.289 INFO:teuthology.orchestra.run.smithi026.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 175 kB/s | 172 kB 00:00 2024-04-23T18:04:41.897 INFO:teuthology.orchestra.run.smithi026.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-04-23T18:04:42.679 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-04-23T18:04:42.680 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-04-23T18:04:42.864 INFO:teuthology.orchestra.run.smithi026.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 687 kB/s | 5.8 MB 00:08 2024-04-23T18:04:43.535 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:04:43.575 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:04:43.697 INFO:teuthology.orchestra.run.smithi026.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 301 kB/s | 250 kB 00:00 2024-04-23T18:04:44.116 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:04:44.171 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:04:44.260 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:04:44.260 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:04:44.260 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:04:44.260 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:44.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:04:44.598 INFO:teuthology.orchestra.run.smithi026.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 247 kB/s | 222 kB 00:00 2024-04-23T18:04:45.073 INFO:teuthology.orchestra.run.smithi026.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 700 kB/s | 6.0 MB 00:08 2024-04-23T18:04:45.365 INFO:teuthology.orchestra.run.smithi026.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 101 kB/s | 77 kB 00:00 2024-04-23T18:04:45.715 INFO:teuthology.orchestra.run.smithi026.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 66 kB/s | 42 kB 00:00 2024-04-23T18:04:46.238 INFO:teuthology.orchestra.run.smithi026.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 308 kB/s | 269 kB 00:00 2024-04-23T18:04:46.574 INFO:teuthology.orchestra.run.smithi026.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 185 kB/s | 159 kB 00:00 2024-04-23T18:04:47.216 INFO:teuthology.orchestra.run.smithi026.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 285 kB/s | 279 kB 00:00 2024-04-23T18:04:47.241 INFO:teuthology.orchestra.run.smithi026.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-04-23T18:04:47.866 INFO:teuthology.orchestra.run.smithi026.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 83 kB/s | 54 kB 00:00 2024-04-23T18:04:48.508 INFO:teuthology.orchestra.run.smithi026.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-04-23T18:04:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-04-23T18:04:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:04:48.721 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:04:48.850 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:04:48.992 INFO:teuthology.orchestra.run.smithi026.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 736 kB/s | 5.1 MB 00:07 2024-04-23T18:04:49.000 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:04:49.026 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:04:49.069 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:04:49.142 INFO:teuthology.orchestra.run.smithi026.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 70 kB/s | 44 kB 00:00 2024-04-23T18:04:49.399 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:04:49.510 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:04:49.632 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:04:49.662 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:04:49.753 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:04:49.759 INFO:teuthology.orchestra.run.smithi026.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 58 kB/s | 36 kB 00:00 2024-04-23T18:04:49.791 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:04:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 241 kB/s | 221 kB 00:00 2024-04-23T18:04:49.938 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:04:49.996 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:04:50.025 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:04:50.128 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:04:50.150 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:04:50.213 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:04:50.235 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:04:50.256 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:04:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:04:50.485 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:04:50.584 INFO:teuthology.orchestra.run.smithi026.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 57 kB/s | 47 kB 00:00 2024-04-23T18:04:50.592 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:04:50.679 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:04:50.718 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:04:50.753 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:04:50.825 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:04:50.868 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:04:50.912 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:04:50.945 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:04:50.973 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:04:51.026 INFO:teuthology.orchestra.run.smithi026.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 273 kB/s | 305 kB 00:01 2024-04-23T18:04:51.041 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:04:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:04:51.126 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:04:51.161 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:04:51.237 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:04:51.272 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:04:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:04:51.326 INFO:teuthology.orchestra.run.smithi026.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 78 kB/s | 58 kB 00:00 2024-04-23T18:04:51.344 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:04:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:04:51.459 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:04:51.533 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:04:51.665 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:04:51.719 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:04:51.868 INFO:teuthology.orchestra.run.smithi026.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 111 kB/s | 94 kB 00:00 2024-04-23T18:04:52.152 INFO:teuthology.orchestra.run.smithi026.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 77 kB/s | 64 kB 00:00 2024-04-23T18:04:52.236 INFO:teuthology.orchestra.run.smithi026.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-04-23T18:04:52.240 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:04:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:04:52.361 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:04:52.386 INFO:teuthology.orchestra.run.smithi026.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-04-23T18:04:52.404 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:04:52.443 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:04:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:04:52.640 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:04:52.696 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:04:52.702 INFO:teuthology.orchestra.run.smithi026.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 54 kB/s | 45 kB 00:00 2024-04-23T18:04:52.750 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:04:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:04:52.867 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:04:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:04:52.946 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:04:52.991 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:04:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:04:53.076 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:04:53.212 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:04:53.257 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:04:53.287 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:04:53.329 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:04:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:04:53.463 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:04:53.494 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:04:53.520 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:04:53.545 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:04:53.619 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:04:53.619 INFO:teuthology.orchestra.run.smithi026.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 337 kB/s | 309 kB 00:00 2024-04-23T18:04:53.678 INFO:teuthology.orchestra.run.smithi026.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 424 kB/s | 548 kB 00:01 2024-04-23T18:04:53.728 INFO:teuthology.orchestra.run.smithi026.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-04-23T18:04:53.746 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:04:53.778 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:04:53.806 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:04:53.837 INFO:teuthology.orchestra.run.smithi026.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-23T18:04:53.840 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:04:53.879 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:04:53.911 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:04:53.937 INFO:teuthology.orchestra.run.smithi026.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-04-23T18:04:53.941 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:04:53.975 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:04:54.009 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:04:54.050 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:04:54.085 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:04:54.129 INFO:teuthology.orchestra.run.smithi026.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-04-23T18:04:54.171 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:04:54.215 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:04:54.246 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:04:54.294 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:04:54.396 INFO:teuthology.orchestra.run.smithi026.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-04-23T18:04:54.400 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:04:54.448 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:04:55.168 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:04:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:04:55.254 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:04:55.353 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:04:55.404 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:04:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-04-23T18:04:55.508 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-04-23T18:04:55.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:55.532 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:04:55.588 INFO:teuthology.orchestra.run.smithi026.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 703 kB/s | 838 kB 00:01 2024-04-23T18:04:55.676 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:04:55.690 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:04:55.691 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:04:55.691 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:04:55.691 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:04:55.738 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:04:55.815 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:04:56.451 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:04:56.471 INFO:teuthology.orchestra.run.smithi026.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 620 kB/s | 548 kB 00:00 2024-04-23T18:04:56.561 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:04:56.590 INFO:teuthology.orchestra.run.smithi026.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 366 kB/s | 43 kB 00:00 2024-04-23T18:04:56.699 INFO:teuthology.orchestra.run.smithi026.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-23T18:04:56.807 INFO:teuthology.orchestra.run.smithi026.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-23T18:04:57.091 INFO:teuthology.orchestra.run.smithi026.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 606 kB/s | 172 kB 00:00 2024-04-23T18:04:57.179 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:04:57.204 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:04:57.204 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:04:57.204 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:04:57.204 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:04:57.204 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:04:57.205 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:57.456 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:04:57.478 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:04:57.478 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:04:57.478 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:04:57.478 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:04:57.479 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:04:57.479 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:57.633 INFO:teuthology.orchestra.run.smithi026.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 662 kB/s | 358 kB 00:00 2024-04-23T18:04:57.796 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:04:57.886 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:04:57.914 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:04:57.923 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:04:57.945 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:04:57.945 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:04:57.945 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:04:57.945 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:04:57.945 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:04:57.946 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:04:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:04:58.053 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:04:58.100 INFO:teuthology.orchestra.run.smithi026.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 649 kB/s | 303 kB 00:00 2024-04-23T18:04:58.110 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:04:58.166 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:04:58.209 INFO:teuthology.orchestra.run.smithi026.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-23T18:04:58.274 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:04:58.350 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:04:58.399 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:04:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:04:58.477 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:04:58.566 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:04:58.584 INFO:teuthology.orchestra.run.smithi026.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 548 kB/s | 206 kB 00:00 2024-04-23T18:04:58.692 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:04:58.715 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:04:58.723 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:04:58.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:04:58.759 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:04:58.790 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:04:58.818 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:04:59.194 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:04:59.233 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:04:59.237 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:04:59.263 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:04:59.265 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:04:59.326 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:04:59.334 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:04:59.365 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:04:59.383 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:04:59.418 INFO:teuthology.orchestra.run.smithi026.stdout:(121/152): libarrow-9.0.0-10.el9.x86_64.rpm 788 kB/s | 4.4 MB 00:05 2024-04-23T18:04:59.676 INFO:teuthology.orchestra.run.smithi026.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 560 kB/s | 144 kB 00:00 2024-04-23T18:04:59.776 INFO:teuthology.orchestra.run.smithi026.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-23T18:04:59.885 INFO:teuthology.orchestra.run.smithi026.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-23T18:04:59.993 INFO:teuthology.orchestra.run.smithi026.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-04-23T18:05:00.085 INFO:teuthology.orchestra.run.smithi026.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-23T18:05:00.186 INFO:teuthology.orchestra.run.smithi026.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-23T18:05:00.426 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:05:00.466 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:01.144 INFO:teuthology.orchestra.run.smithi026.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 815 kB/s | 2.0 MB 00:02 2024-04-23T18:05:01.236 INFO:teuthology.orchestra.run.smithi026.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-04-23T18:05:01.411 INFO:teuthology.orchestra.run.smithi026.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 451 kB/s | 79 kB 00:00 2024-04-23T18:05:01.431 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:01.458 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:01.528 INFO:teuthology.orchestra.run.smithi026.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 779 kB/s | 1.0 MB 00:01 2024-04-23T18:05:01.559 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:01.629 INFO:teuthology.orchestra.run.smithi026.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 266 kB/s | 58 kB 00:00 2024-04-23T18:05:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:01.695 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:01.722 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:05:01.729 INFO:teuthology.orchestra.run.smithi026.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-23T18:05:01.743 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:01.766 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:01.811 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:01.821 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:01.912 INFO:teuthology.orchestra.run.smithi026.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 490 kB/s | 90 kB 00:00 2024-04-23T18:05:01.919 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:05:01.971 INFO:teuthology.orchestra.run.smithi026.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 616 kB/s | 272 kB 00:00 2024-04-23T18:05:01.976 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:05:02.013 INFO:teuthology.orchestra.run.smithi026.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-04-23T18:05:02.072 INFO:teuthology.orchestra.run.smithi026.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-23T18:05:02.236 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:02.272 INFO:teuthology.orchestra.run.smithi026.stdout:(138/152): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-04-23T18:05:02.274 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:05:02.298 INFO:teuthology.orchestra.run.smithi026.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 664 kB/s | 188 kB 00:00 2024-04-23T18:05:02.365 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:05:02.365 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:05:02.365 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:05:02.365 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:02.392 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:05:02.397 INFO:teuthology.orchestra.run.smithi026.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-04-23T18:05:02.423 INFO:teuthology.orchestra.run.smithi026.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-04-23T18:05:02.656 INFO:teuthology.orchestra.run.smithi026.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 384 kB/s | 90 kB 00:00 2024-04-23T18:05:02.832 INFO:teuthology.orchestra.run.smithi026.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 529 kB/s | 230 kB 00:00 2024-04-23T18:05:02.906 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:05:02.939 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:02.961 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:02.965 INFO:teuthology.orchestra.run.smithi026.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-04-23T18:05:03.006 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:03.028 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:03.028 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:03.028 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:05:03.028 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:03.073 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:03.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:03.341 INFO:teuthology.orchestra.run.smithi026.stdout:(145/152): re2-20211101-3.el9.x86_64.rpm 511 kB/s | 192 kB 00:00 2024-04-23T18:05:03.366 INFO:teuthology.orchestra.run.smithi026.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 602 kB/s | 427 kB 00:00 2024-04-23T18:05:03.407 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:03.430 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:03.430 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:03.430 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:05:03.430 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:03.431 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:03.431 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:03.691 INFO:teuthology.orchestra.run.smithi026.stdout:(147/152): librados2-19.0.0-3156.gd69f13d8.el7. 10 MB/s | 3.4 MB 00:00 2024-04-23T18:05:03.900 INFO:teuthology.orchestra.run.smithi026.stdout:(148/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 15 MB/s | 3.1 MB 00:00 2024-04-23T18:05:04.233 INFO:teuthology.orchestra.run.smithi026.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 1.3 MB 00:00 2024-04-23T18:05:04.542 INFO:teuthology.orchestra.run.smithi026.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.8 MB/s | 2.4 MB 00:00 2024-04-23T18:05:05.334 INFO:teuthology.orchestra.run.smithi026.stdout:(151/152): thrift-0.15.0-2.el9.x86_64.rpm 813 kB/s | 1.6 MB 00:01 2024-04-23T18:05:08.044 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:05:08.085 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:05:08.126 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:05:08.158 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:05:08.186 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:05:08.370 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:05:08.415 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:05:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:05:09.086 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:09.118 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:09.137 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:09.965 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:05:10.025 INFO:teuthology.orchestra.run.smithi026.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 730 kB/s | 16 MB 00:22 2024-04-23T18:05:10.031 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:05:10.031 INFO:teuthology.orchestra.run.smithi026.stdout:Total 4.0 MB/s | 222 MB 00:55 2024-04-23T18:05:10.056 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:05:10.097 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:05:10.180 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:05:10.213 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:05:10.213 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:05:10.296 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:05:10.321 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:05:10.321 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:05:10.412 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:05:10.477 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:11.138 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-23T18:05:11.297 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-23T18:05:11.297 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-23T18:05:13.167 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-23T18:05:13.168 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-23T18:05:13.742 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:05:13.742 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /sys 2024-04-23T18:05:13.742 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /proc 2024-04-23T18:05:13.742 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /mnt 2024-04-23T18:05:13.742 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /var/tmp 2024-04-23T18:05:13.743 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /home 2024-04-23T18:05:13.743 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /root 2024-04-23T18:05:13.743 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /tmp 2024-04-23T18:05:13.743 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:13.917 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:05:13.934 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:05:13.934 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:13.934 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:05:13.934 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:15.428 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:05:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:05:15.454 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:15.455 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:05:15.455 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:05:15.455 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:05:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:15.679 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:05:15.701 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:05:15.702 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:15.702 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:05:15.702 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:05:15.702 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:05:15.702 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:16.147 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:05:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:16.293 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:05:16.342 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:05:16.398 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:05:16.582 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:05:16.639 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:05:16.706 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:05:16.721 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:16.721 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:05:16.721 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:05:16.721 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:05:16.721 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:05:16.722 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:05:16.723 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:05:16.724 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:05:16.725 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:05:16.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:05:16.727 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:05:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:05:16.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:05:16.730 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:05:16.731 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:05:16.732 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:05:16.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:05:16.787 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:05:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:05:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:05:17.022 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:05:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:05:17.503 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:05:17.532 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:05:17.564 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:05:17.625 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:05:17.655 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:05:18.692 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:05:18.732 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:18.798 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.799 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.800 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:05:18.801 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.802 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:05:18.803 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:05:18.804 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:05:18.805 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:05:18.806 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:05:18.807 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:05:18.808 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:05:18.809 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:05:18.810 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:05:18.810 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-23T18:05:18.810 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-04-23T18:05:19.189 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-23T18:05:19.203 DEBUG:teuthology.parallel:result is None 2024-04-23T18:05:19.270 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:05:19.325 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:05:19.380 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:05:19.430 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:05:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:05:19.498 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:05:19.697 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:19.724 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:19.827 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:19.844 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:05:19.955 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:05:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:05:20.018 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:20.041 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:20.042 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:05:20.042 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:20.042 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:20.080 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:05:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:20.107 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:05:20.194 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:05:20.198 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:05:20.227 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:05:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:05:20.384 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:05:20.433 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:05:20.460 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:05:20.549 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:05:20.578 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:05:20.641 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:05:20.662 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:05:20.684 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:05:20.880 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:05:20.912 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:05:21.024 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:05:21.118 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:05:21.153 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:05:21.172 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:05:21.188 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:05:21.205 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:21.228 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:21.264 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:05:21.272 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:21.295 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:21.295 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:21.295 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:05:21.295 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:21.304 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:05:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:21.348 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:05:21.362 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:21.363 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:21.363 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:05:21.363 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:21.363 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:21.363 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:21.380 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:05:21.408 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:05:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:05:21.539 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:05:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:05:21.604 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:05:21.665 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:21.683 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:05:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:21.688 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:21.688 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:05:21.688 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:21.689 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:05:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:05:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:05:22.034 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:05:22.105 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:05:22.187 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:05:22.331 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:05:22.380 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:05:22.943 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:05:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:05:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:05:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:05:23.134 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:05:23.177 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:05:23.328 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:05:23.379 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:05:23.435 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:05:23.508 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:05:23.546 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:05:23.582 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:05:23.630 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:05:23.665 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:05:23.709 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:05:23.750 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:05:23.894 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:05:23.939 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:05:23.969 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:05:24.011 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:05:24.110 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:05:24.152 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:05:24.192 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:05:24.218 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:05:24.242 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:05:24.320 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:05:24.450 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:05:24.483 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:05:24.511 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:05:24.546 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:05:24.585 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:05:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:05:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:05:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:05:24.722 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:05:24.764 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:05:24.798 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:05:24.888 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:05:24.937 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:05:24.968 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:05:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:05:25.127 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:05:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:05:25.938 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:05:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:05:26.025 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:05:26.127 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:05:26.184 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:05:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:05:26.286 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:05:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:05:26.326 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:05:26.367 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:05:26.400 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:05:26.428 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:05:26.497 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:05:26.533 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:05:26.607 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:05:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:05:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:05:26.679 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:05:27.297 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:05:27.300 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:27.335 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:27.355 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:27.418 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:05:28.182 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:05:28.267 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:05:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:05:28.399 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:05:28.438 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:05:28.438 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:05:28.522 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:05:28.555 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:05:28.555 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:05:28.648 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:05:28.712 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:28.720 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:05:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:05:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:05:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:05:29.186 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:05:29.242 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:05:29.335 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:05:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:05:29.401 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:05:29.910 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:05:29.938 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:05:30.009 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:05:30.040 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:05:32.399 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:32.436 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:33.007 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:05:33.047 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:05:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:05:33.137 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:05:33.137 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:05:33.137 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:33.174 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:05:34.713 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:05:34.714 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:05:34.715 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:05:34.715 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:05:34.715 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:05:34.727 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:05:34.728 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:05:34.729 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:05:34.730 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:05:34.731 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:05:34.732 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:05:34.733 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:05:34.734 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:05:34.735 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:05:34.736 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:05:34.737 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout:Upgraded: 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:36.785 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.787 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:05:36.788 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.789 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:05:36.790 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.791 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:05:36.792 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:05:36.793 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.794 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:05:36.795 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:05:36.796 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:05:36.797 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:05:36.797 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:05:36.797 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-04-23T18:05:37.179 DEBUG:teuthology.parallel:result is None 2024-04-23T18:05:45.172 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-04-23T18:05:45.173 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-04-23T18:05:45.174 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-04-23T18:05:45.174 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:45.351 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:05:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:05:45.368 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:45.368 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:05:45.368 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:46.922 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:05:46.949 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:47.181 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:05:47.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:47.673 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:05:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:47.804 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:05:47.862 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:05:47.926 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:05:48.121 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:05:48.166 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:05:48.239 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:05:48.337 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:05:48.483 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:05:48.524 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:05:48.556 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:05:48.583 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:05:49.054 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:05:49.082 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:05:49.114 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:05:49.178 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:05:49.214 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:05:50.315 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:05:50.356 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:51.380 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:05:51.415 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:51.523 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:05:51.695 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:05:51.725 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:51.749 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:05:51.749 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:51.749 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:05:51.749 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:51.749 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:05:51.750 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:51.801 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:51.812 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:05:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:05:51.967 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:05:52.949 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:05:52.986 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:53.010 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:05:53.011 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:53.011 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:05:53.011 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:53.011 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:05:53.011 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:53.054 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:05:53.077 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:53.077 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:05:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:53.129 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:05:53.153 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:53.472 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:05:53.495 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:05:53.496 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:05:53.496 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:53.496 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:05:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:05:58.375 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:05:58.417 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:05:58.458 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:05:58.489 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:05:58.516 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:05:58.726 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:05:58.771 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:05:58.800 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:05:59.386 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:59.423 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:05:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:06:00.444 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:06:00.486 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:06:00.570 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:06:00.598 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:06:00.599 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:00.674 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:00.707 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:00.707 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:00.809 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:00.874 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:06:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:06:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:06:06.681 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:06:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:06:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:06:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:06:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:06:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:06:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:06:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:06:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:06:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:06:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:06:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:06:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:06:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:06:08.912 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-04-23T18:06:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:06:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:06:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:06:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:06:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:06:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:06:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:06:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:06:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:06:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:06:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:06:08.924 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-23T18:06:09.324 DEBUG:teuthology.parallel:result is None 2024-04-23T18:06:09.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:09.548 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:06:09.570 INFO:teuthology.orchestra.run.smithi026.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.571 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.571 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:06:09.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:09.724 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:06:09.750 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.750 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.750 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:06:09.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:09.973 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:06:09.999 INFO:teuthology.orchestra.run.smithi131.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.999 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:06:09.999 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:06:10.001 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T18:06:10.001 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:06:10.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:06:10.028 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:06:10.028 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:06:10.059 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:06:10.059 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:06:10.089 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T18:06:10.089 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:06:10.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:06:10.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:06:10.178 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:06:10.178 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:06:10.206 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:06:10.273 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:06:10.273 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:06:10.302 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:06:10.370 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T18:06:10.371 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:06:10.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:06:10.397 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:06:10.463 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:06:10.463 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:06:10.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:06:10.555 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:06:10.555 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:06:10.579 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:06:10.644 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T18:06:10.644 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:06:10.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:06:10.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:06:10.735 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:06:10.735 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:06:10.760 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:06:10.824 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:06:10.824 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:06:10.847 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:06:10.914 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T18:06:11.027 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:06:11.028 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:11.028 INFO:tasks.cephadm:Cluster fsid is 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:06:11.029 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T18:06:11.029 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.26', 'mon.b': '172.21.15.110', 'mon.c': '172.21.15.131'} 2024-04-23T18:06:11.029 INFO:tasks.cephadm:First mon is mon.a on smithi026 2024-04-23T18:06:11.029 INFO:tasks.cephadm:First mgr is x 2024-04-23T18:06:11.029 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T18:06:11.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname $(hostname -s) 2024-04-23T18:06:11.055 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-04-23T18:06:11.079 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2024-04-23T18:06:11.102 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T18:06:11.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:11.244 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T18:06:11.492 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:06:11.495 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-23T18:06:11.495 DEBUG:teuthology.orchestra.run.smithi026:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:12.144 INFO:teuthology.orchestra.run.smithi026.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:06 /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:12.145 DEBUG:teuthology.orchestra.run.smithi110:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:12.736 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:06 /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:12.737 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:13.596 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:06 /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:13.597 DEBUG:teuthology.orchestra.run.smithi026:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:13.616 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:13.635 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:06:13.670 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-23T18:06:13.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:06:13.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:06:13.677 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:06:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:06:15.246 INFO:teuthology.orchestra.run.smithi026.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:06:15.411 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:06:55.217 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "repo_digests": [ 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:06:55.218 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:06:55.365 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-23T18:06:55.366 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-23T18:06:56.654 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "repo_digests": [ 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-04-23T18:06:56.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-04-23T18:06:56.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph 2024-04-23T18:06:56.713 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-04-23T18:06:56.744 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2024-04-23T18:06:56.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /etc/ceph 2024-04-23T18:06:56.800 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-04-23T18:06:56.827 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2024-04-23T18:06:56.852 INFO:tasks.cephadm:Writing seed config... 2024-04-23T18:06:56.853 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-23T18:06:56.853 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-23T18:06:56.853 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-23T18:06:56.853 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-23T18:06:56.854 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T18:06:56.855 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-23T18:06:56.856 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T18:06:56.857 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-23T18:06:56.858 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:06:56.858 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T18:06:56.876 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2a701120-019c-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-23T18:06:56.876 DEBUG:teuthology.orchestra.run.smithi026:mon.a> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T18:06:56.919 DEBUG:teuthology.orchestra.run.smithi026:mgr.x> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T18:06:56.963 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T18:06:56.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid 2a701120-019c-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.26 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:06:57.165 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:06:57.165 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', '2a701120-019c-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.26', '--skip-admin-label'] 2024-04-23T18:06:57.166 INFO:teuthology.orchestra.run.smithi026.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T18:06:57.166 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying podman|docker is present... 2024-04-23T18:06:57.192 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:06:57.192 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying lvm2 is present... 2024-04-23T18:06:57.192 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying time synchronization is in place... 2024-04-23T18:06:57.200 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:06:57.200 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:06:57.206 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:06:57.206 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:06:57.213 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-04-23T18:06:57.219 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-04-23T18:06:57.219 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:06:57.219 INFO:teuthology.orchestra.run.smithi026.stdout:Repeating the final host check... 2024-04-23T18:06:57.244 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:06:57.244 INFO:teuthology.orchestra.run.smithi026.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T18:06:57.245 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl is present 2024-04-23T18:06:57.245 INFO:teuthology.orchestra.run.smithi026.stdout:lvcreate is present 2024-04-23T18:06:57.251 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:06:57.251 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:06:57.257 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:06:57.257 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:06:57.263 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:Host looks OK 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:Cluster fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:Acquiring lock 139938937843664 on /run/cephadm/2a701120-019c-11ef-bc93-c7b262605968.lock 2024-04-23T18:06:57.270 INFO:teuthology.orchestra.run.smithi026.stdout:Lock 139938937843664 acquired on /run/cephadm/2a701120-019c-11ef-bc93-c7b262605968.lock 2024-04-23T18:06:57.271 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 3300 ... 2024-04-23T18:06:57.271 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 6789 ... 2024-04-23T18:06:57.271 INFO:teuthology.orchestra.run.smithi026.stdout:Base mon IP(s) is [172.21.15.26:3300, 172.21.15.26:6789], mon addrv is [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-04-23T18:06:57.274 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.26 metric 100 2024-04-23T18:06:57.274 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.26 metric 100 2024-04-23T18:06:57.277 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T18:06:57.277 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-23T18:06:57.277 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-23T18:06:57.280 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T18:06:57.280 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T18:06:57.281 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:06:57.281 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-23T18:06:57.281 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-04-23T18:06:57.281 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:06:57.282 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-04-23T18:06:57.282 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-04-23T18:06:57.282 INFO:teuthology.orchestra.run.smithi026.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-23T18:06:57.282 INFO:teuthology.orchestra.run.smithi026.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T18:06:57.283 INFO:teuthology.orchestra.run.smithi026.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:07:00.762 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:07:00.763 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T18:07:06.912 INFO:teuthology.orchestra.run.smithi026.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:07:06.912 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:07:06.912 INFO:teuthology.orchestra.run.smithi026.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T18:07:07.782 INFO:teuthology.orchestra.run.smithi026.stdout:stat: stdout 167 167 2024-04-23T18:07:07.782 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial keys... 2024-04-23T18:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDM+Cdm4VCUBxAAIURwGqUnmv75lDR3Co/Y4A== 2024-04-23T18:07:09.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDN+CdmZubZABAAbbHDlo3HpfOdL8/+LL539g== 2024-04-23T18:07:10.497 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDN+Cdm2sgoNBAAqvwFW3Hh9xtcKmvXphUJTg== 2024-04-23T18:07:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial monmap... 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool for a [v2:172.21.15.26:3300,v1:172.21.15.26:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:07:11.463 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-04-23T18:07:11.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: set fsid to 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:11.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:07:11.464 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:07:11.464 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mon... 2024-04-23T18:07:15.064 INFO:teuthology.orchestra.run.smithi026.stdout:create mon.a on 2024-04-23T18:07:15.278 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-23T18:07:15.429 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T18:07:15.617 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2a701120-019c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968.target. 2024-04-23T18:07:15.617 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2a701120-019c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968.target. 2024-04-23T18:07:15.834 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:07:15.834 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T18:07:16.002 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968.target.wants/ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968@.service. 2024-04-23T18:07:16.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:16 smithi026 ceph-mon[28992]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:07:16.529 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:07:16.529 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-23T18:07:16.536 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:07:16.536 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:07:16.536 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-23T18:07:16.537 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:07:16.537 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon to start... 2024-04-23T18:07:16.537 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon... 2024-04-23T18:07:16.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:16 smithi026 bash[28973]: 9b9d10535cf74aeacc72ab85d85b25645a145858a0aeb376b744bef832013663 2024-04-23T18:07:16.769 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:16 smithi026 systemd[1]: Started Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: monmap epoch 1 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: last_changed 2024-04-23T18:07:10.862733+0000 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: min_mon_release 19 (squid) 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: election_strategy: 1 2024-04-23T18:07:17.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:07:17.658 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: fsmap 2024-04-23T18:07:17.658 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:07:17.658 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: mgrmap e1: no daemons active 2024-04-23T18:07:17.658 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:17 smithi026 ceph-mon[28992]: from='client.? 172.21.15.26:0/1621747720' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout id: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout services: 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.576251s) 2024-04-23T18:07:17.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout data: 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.976+0000 7fc85bad9700 1 Processor -- start 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.976+0000 7fc85bad9700 1 -- start start 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.976+0000 7fc85bad9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.976+0000 7fc85bad9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc85407be20 con 0x7fc854106e40 2024-04-23T18:07:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.976+0000 7fc859875700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc859875700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41110/0 (socket says 172.21.15.26:41110) 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc859875700 1 -- 172.21.15.26:0/2301363168 learned_addr learned my addr 172.21.15.26:0/2301363168 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc859875700 1 -- 172.21.15.26:0/2301363168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8541077a0 con 0x7fc854106e40 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc859875700 1 --2- 172.21.15.26:0/2301363168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc8500098a0 tx=0x7fc850009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77769bdfba0e7189 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc858873700 1 -- 172.21.15.26:0/2301363168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc850004160 con 0x7fc854106e40 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc858873700 1 -- 172.21.15.26:0/2301363168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc850004300 con 0x7fc854106e40 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.977+0000 7fc858873700 1 -- 172.21.15.26:0/2301363168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8500036a0 con 0x7fc854106e40 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 -- 172.21.15.26:0/2301363168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 msgr2=0x7fc854107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 --2- 172.21.15.26:0/2301363168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc8500098a0 tx=0x7fc850009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 -- 172.21.15.26:0/2301363168 shutdown_connections 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 --2- 172.21.15.26:0/2301363168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc854107260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 -- 172.21.15.26:0/2301363168 >> 172.21.15.26:0/2301363168 conn(0x7fc854102a40 msgr2=0x7fc854104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:17.668 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 -- 172.21.15.26:0/2301363168 shutdown_connections 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 -- 172.21.15.26:0/2301363168 wait complete. 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.978+0000 7fc85bad9700 1 Processor -- start 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc85bad9700 1 -- start start 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc85bad9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc85bad9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc854107cd0 con 0x7fc854106e40 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc859875700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc859875700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41126/0 (socket says 172.21.15.26:41126) 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc859875700 1 -- 172.21.15.26:0/1621747720 learned_addr learned my addr 172.21.15.26:0/1621747720 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc859875700 1 -- 172.21.15.26:0/1621747720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85419e320 con 0x7fc854106e40 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc859875700 1 --2- 172.21.15.26:0/1621747720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc850004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc850003db0 con 0x7fc854106e40 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc850011630 con 0x7fc854106e40 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85419e5b0 con 0x7fc854106e40 2024-04-23T18:07:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.979+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc850011950 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.980+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8541a12a0 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.980+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc850011af0 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.980+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc85001c450 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.980+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8540640a0 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:16.983+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc850032020 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.056+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fc854197340 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.056+0000 7fc84affd700 1 -- 172.21.15.26:0/1621747720 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fc854197340 con 0x7fc854106e40 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 msgr2=0x7fc85419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 --2- 172.21.15.26:0/1621747720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc850004750 comp rx=0 tx=0).stop 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 shutdown_connections 2024-04-23T18:07:17.670 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 --2- 172.21.15.26:0/1621747720 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc854106e40 0x7fc85419dde0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 >> 172.21.15.26:0/1621747720 conn(0x7fc854102a40 msgr2=0x7fc854194400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 shutdown_connections 2024-04-23T18:07:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:17.057+0000 7fc85bad9700 1 -- 172.21.15.26:0/1621747720 wait complete. 2024-04-23T18:07:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:mon is available 2024-04-23T18:07:17.671 INFO:teuthology.orchestra.run.smithi026.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T18:07:18.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:18 smithi026 ceph-mon[28992]: from='client.? 172.21.15.26:0/1087891007' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:07:18.898 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:18 smithi026 ceph-mon[28992]: from='client.? 172.21.15.26:0/1087891007' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T18:07:18.909 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:18.909 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:07:18.909 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:07:18.910 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:07:18.911 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.089+0000 7f1defe1b700 1 Processor -- start 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.089+0000 7f1defe1b700 1 -- start start 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.089+0000 7f1defe1b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.089+0000 7f1defe1b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1de8063d00 con 0x7f1de8109010 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1dedbb7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1dedbb7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55034/0 (socket says 172.21.15.26:55034) 2024-04-23T18:07:18.912 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1dedbb7700 1 -- 172.21.15.26:0/69923604 learned_addr learned my addr 172.21.15.26:0/69923604 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1dedbb7700 1 -- 172.21.15.26:0/69923604 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de810b940 con 0x7f1de8109010 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1dedbb7700 1 --2- 172.21.15.26:0/69923604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80098a0 tx=0x7f1dd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f090dd87ab402890 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1decbb5700 1 -- 172.21.15.26:0/69923604 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd8004160 con 0x7f1de8109010 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1decbb5700 1 -- 172.21.15.26:0/69923604 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd8004300 con 0x7f1de8109010 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.090+0000 7f1decbb5700 1 -- 172.21.15.26:0/69923604 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd80036a0 con 0x7f1de8109010 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 -- 172.21.15.26:0/69923604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 msgr2=0x7f1de810b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 --2- 172.21.15.26:0/69923604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80098a0 tx=0x7f1dd8009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 -- 172.21.15.26:0/69923604 shutdown_connections 2024-04-23T18:07:18.913 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 --2- 172.21.15.26:0/69923604 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de810b400 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 -- 172.21.15.26:0/69923604 >> 172.21.15.26:0/69923604 conn(0x7f1de8102a40 msgr2=0x7f1de8104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 -- 172.21.15.26:0/69923604 shutdown_connections 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.091+0000 7f1defe1b700 1 -- 172.21.15.26:0/69923604 wait complete. 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1defe1b700 1 Processor -- start 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1defe1b700 1 -- start start 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1defe1b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1defe1b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1de810bdb0 con 0x7f1de8109010 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1dedbb7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:18.914 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1dedbb7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55042/0 (socket says 172.21.15.26:55042) 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1dedbb7700 1 -- 172.21.15.26:0/1087891007 learned_addr learned my addr 172.21.15.26:0/1087891007 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1dedbb7700 1 -- 172.21.15.26:0/1087891007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de819c160 con 0x7f1de8109010 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1dedbb7700 1 --2- 172.21.15.26:0/1087891007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd8003db0 con 0x7f1de8109010 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd8011630 con 0x7f1de8109010 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.092+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de819c3f0 con 0x7f1de8109010 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.093+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd8011950 con 0x7f1de8109010 2024-04-23T18:07:18.915 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.093+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de819f0e0 con 0x7f1de8109010 2024-04-23T18:07:18.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.093+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1dd8011af0 con 0x7f1de8109010 2024-04-23T18:07:18.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.093+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1dd801c450 con 0x7f1de8109010 2024-04-23T18:07:18.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.093+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de80640a0 con 0x7f1de8109010 2024-04-23T18:07:18.916 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.096+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1dd8032020 con 0x7f1de8109010 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.166+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1de802e180 con 0x7f1de8109010 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.169+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7f1dd8004260 con 0x7f1de8109010 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.170+0000 7f1ddeffd700 1 -- 172.21.15.26:0/1087891007 <== mon.0 v2:172.21.15.26:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1dd801c6a0 con 0x7f1de8109010 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 msgr2=0x7f1de819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 --2- 172.21.15.26:0/1087891007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd8004750 comp rx=0 tx=0).stop 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 shutdown_connections 2024-04-23T18:07:18.917 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 --2- 172.21.15.26:0/1087891007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1de8109010 0x7f1de819bc20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:18.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 >> 172.21.15.26:0/1087891007 conn(0x7f1de8102a40 msgr2=0x7f1de8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:18.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 shutdown_connections 2024-04-23T18:07:18.918 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:18.171+0000 7f1defe1b700 1 -- 172.21.15.26:0/1087891007 wait complete. 2024-04-23T18:07:18.918 INFO:teuthology.orchestra.run.smithi026.stdout:Generating new minimal ceph.conf... 2024-04-23T18:07:19.830 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:19 smithi026 ceph-mon[28992]: from='client.? 172.21.15.26:0/1902611768' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:07:19.837 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5afa80700 1 Processor -- start 2024-04-23T18:07:19.837 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5afa80700 1 -- start start 2024-04-23T18:07:19.837 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5afa80700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5afa80700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a8108f40 con 0x7fc5a8108550 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5ad81c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5ad81c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55058/0 (socket says 172.21.15.26:55058) 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5ad81c700 1 -- 172.21.15.26:0/2459329274 learned_addr learned my addr 172.21.15.26:0/2459329274 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.209+0000 7fc5ad81c700 1 -- 172.21.15.26:0/2459329274 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a8109700 con 0x7fc5a8108550 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5ad81c700 1 --2- 172.21.15.26:0/2459329274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc5980098a0 tx=0x7fc598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3f72e18c245f2d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5ac81a700 1 -- 172.21.15.26:0/2459329274 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc598004160 con 0x7fc5a8108550 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5ac81a700 1 -- 172.21.15.26:0/2459329274 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc598004300 con 0x7fc5a8108550 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5ac81a700 1 -- 172.21.15.26:0/2459329274 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc598011af0 con 0x7fc5a8108550 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5afa80700 1 -- 172.21.15.26:0/2459329274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 msgr2=0x7fc5a8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5afa80700 1 --2- 172.21.15.26:0/2459329274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc5980098a0 tx=0x7fc598009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5afa80700 1 -- 172.21.15.26:0/2459329274 shutdown_connections 2024-04-23T18:07:19.838 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5afa80700 1 --2- 172.21.15.26:0/2459329274 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a8108970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.210+0000 7fc5afa80700 1 -- 172.21.15.26:0/2459329274 >> 172.21.15.26:0/2459329274 conn(0x7fc5a8103b10 msgr2=0x7fc5a8105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 -- 172.21.15.26:0/2459329274 shutdown_connections 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 -- 172.21.15.26:0/2459329274 wait complete. 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 Processor -- start 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 -- start start 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5afa80700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810cab0 con 0x7fc5a8108550 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5ad81c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5ad81c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55066/0 (socket says 172.21.15.26:55066) 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.211+0000 7fc5ad81c700 1 -- 172.21.15.26:0/1902611768 learned_addr learned my addr 172.21.15.26:0/1902611768 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc5ad81c700 1 -- 172.21.15.26:0/1902611768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a819ca40 con 0x7fc5a8108550 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc5ad81c700 1 --2- 172.21.15.26:0/1902611768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc5980117b0 tx=0x7fc598011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc598003af0 con 0x7fc5a8108550 2024-04-23T18:07:19.839 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc598020070 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a819ccd0 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc598019630 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a819f9c0 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc598019c30 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.212+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc59800b720 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.213+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a80640a0 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.215+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc59801a9c0 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.286+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc5a819fb90 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.286+0000 7fc59effd700 1 -- 172.21.15.26:0/1902611768 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fc59801a430 con 0x7fc5a8108550 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 msgr2=0x7fc5a819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 --2- 172.21.15.26:0/1902611768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc5980117b0 tx=0x7fc598011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 shutdown_connections 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 --2- 172.21.15.26:0/1902611768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5a8108550 0x7fc5a819c500 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 >> 172.21.15.26:0/1902611768 conn(0x7fc5a8103b10 msgr2=0x7fc5a81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 shutdown_connections 2024-04-23T18:07:19.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:19.287+0000 7fc5afa80700 1 -- 172.21.15.26:0/1902611768 wait complete. 2024-04-23T18:07:19.841 INFO:teuthology.orchestra.run.smithi026.stdout:Restarting the monitor... 2024-04-23T18:07:20.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:19 smithi026 systemd[1]: Stopping Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:07:20.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a[28988]: 2024-04-23T18:07:20.147+0000 7f451d7c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:07:20.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a[28988]: 2024-04-23T18:07:20.147+0000 7f451d7c7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T18:07:20.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 podman[29238]: 2024-04-23 18:07:20.260715345 +0000 UTC m=+0.244671786 container died 9b9d10535cf74aeacc72ab85d85b25645a145858a0aeb376b744bef832013663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T18:07:20.820 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 podman[29238]: 2024-04-23 18:07:20.597610502 +0000 UTC m=+0.581566945 container cleanup 9b9d10535cf74aeacc72ab85d85b25645a145858a0aeb376b744bef832013663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:07:20.821 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 bash[29238]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:07:21.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:20 smithi026 podman[29252]: 2024-04-23 18:07:20.826888735 +0000 UTC m=+0.561168544 container remove 9b9d10535cf74aeacc72ab85d85b25645a145858a0aeb376b744bef832013663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-23T18:07:21.072 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T18:07:21.072 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 systemd[1]: Stopped Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:07:21.072 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 systemd[1]: Starting Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:07:21.402 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 podman[29345]: 2024-04-23 18:07:21.196135246 +0000 UTC m=+0.018208078 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:21.403 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 podman[29345]: 2024-04-23 18:07:21.299154028 +0000 UTC m=+0.121226843 container create d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T18:07:21.608 INFO:teuthology.orchestra.run.smithi026.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T18:07:21.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 podman[29345]: 2024-04-23 18:07:21.526427746 +0000 UTC m=+0.348500561 container init d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:07:21.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 podman[29345]: 2024-04-23 18:07:21.529422716 +0000 UTC m=+0.351495532 container start d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:07:21.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: pidfile_write: ignore empty --pid-file 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: load: jerasure load: lrc 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Git sha 0 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: DB SUMMARY 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: DB Session ID: 09MGQHT24LB5S5U6GVO8 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T18:07:21.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105442 ; 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.env: 0x55aec382fcc0 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.info_log: 0x55aec574fbe0 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:07:21.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.statistics: (nil) 2024-04-23T18:07:21.849 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.use_fsync: 0 2024-04-23T18:07:21.849 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:07:21.849 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:07:21.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.db_log_dir: 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.wal_dir: 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.write_buffer_manager: 0x55aec56e30e0 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:07:21.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.unordered_write: 0 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.row_cache: None 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.wal_filter: None 2024-04-23T18:07:21.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.wal_compression: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:07:21.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:07:21.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:07:21.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_open_files: -1 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:07:21.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Compression algorithms supported: 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kZSTD supported: 0 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:07:21.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.merge_operator: 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_filter: None 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aec574fdc0) 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:07:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_type: 0 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_index_type: 0 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_shortening: 1 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: checksum: 4 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: no_block_cache: 0 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache: 0x55aec57151f0 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_options: 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: capacity : 536870912 2024-04-23T18:07:21.859 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_shard_bits : 4 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: strict_capacity_limit : 0 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_compressed: (nil) 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: persistent_cache: (nil) 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size: 4096 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size_deviation: 10 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_restart_interval: 16 2024-04-23T18:07:21.860 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_block_restart_interval: 1 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: metadata_block_size: 4096 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: partition_filters: 0 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: use_delta_encoding: 1 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: filter_policy: bloomfilter 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: whole_key_filtering: 1 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: verify_compression: 0 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: format_version: 5 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_index_compression: 1 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_align: 0 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: max_auto_readahead_size: 262144 2024-04-23T18:07:21.861 INFO:journalctl@ceph.mon.a.smithi026.stdout: prepopulate_block_cache: 0 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression: NoCompression 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.num_levels: 7 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:07:21.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:07:21.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:07:21.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:07:21.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:07:21.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.ttl: 2592000 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:07:21.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f202c4e9-dc86-45ac-b7c6-58bcd624ed58 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895641559205, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T18:07:21.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895641560235, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100554, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98439, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90693, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895641, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f202c4e9-dc86-45ac-b7c6-58bcd624ed58", "db_session_id": "09MGQHT24LB5S5U6GVO8", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895641560320, "job": 1, "event": "recovery_finished"} 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aec582a000 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: DB pointer 0x55aec5814000 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** DB Stats ** 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:07:21.869 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: L0 2/0 100.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: Sum 2/0 100.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:07:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:07:21.871 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative compaction: 0.00 GB write, 30.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval compaction: 0.00 GB write, 30.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache BinnedLRUCache@0x55aec57151f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: starting mon.a rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???) e1 preinit fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).mds e1 new map 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).mds e1 print_map 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: e1 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: legacy client fscid: -1 2024-04-23T18:07:21.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: No filesystems configured 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 bash[29345]: d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 systemd[1]: Started Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: monmap epoch 1 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: last_changed 2024-04-23T18:07:10.862733+0000 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:07:21.873 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: min_mon_release 19 (squid) 2024-04-23T18:07:21.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: election_strategy: 1 2024-04-23T18:07:21.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:07:21.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: fsmap 2024-04-23T18:07:21.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:07:21.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:21 smithi026 ceph-mon[29364]: mgrmap e1: no daemons active 2024-04-23T18:07:22.718 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.050+0000 7f6925477700 1 Processor -- start 2024-04-23T18:07:22.718 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.050+0000 7f6925477700 1 -- start start 2024-04-23T18:07:22.718 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.050+0000 7f6925477700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:22.718 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.050+0000 7f6925477700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6920108f50 con 0x7f6920108560 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55070/0 (socket says 172.21.15.26:55070) 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691effd700 1 -- 172.21.15.26:0/2369666133 learned_addr learned my addr 172.21.15.26:0/2369666133 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691effd700 1 -- 172.21.15.26:0/2369666133 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6920109710 con 0x7f6920108560 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691effd700 1 --2- 172.21.15.26:0/2369666133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6918009e80 tx=0x7f691800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c846ed0a338244 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691dffb700 1 -- 172.21.15.26:0/2369666133 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6918017070 con 0x7f6920108560 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691dffb700 1 -- 172.21.15.26:0/2369666133 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f691800b8c0 con 0x7f6920108560 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.051+0000 7f691dffb700 1 -- 172.21.15.26:0/2369666133 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6918004490 con 0x7f6920108560 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 -- 172.21.15.26:0/2369666133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 msgr2=0x7f6920108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 --2- 172.21.15.26:0/2369666133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6918009e80 tx=0x7f691800b330 comp rx=0 tx=0).stop 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 -- 172.21.15.26:0/2369666133 shutdown_connections 2024-04-23T18:07:22.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 --2- 172.21.15.26:0/2369666133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f6920108980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 -- 172.21.15.26:0/2369666133 >> 172.21.15.26:0/2369666133 conn(0x7f6920103b40 msgr2=0x7f6920105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 -- 172.21.15.26:0/2369666133 shutdown_connections 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.052+0000 7f6925477700 1 -- 172.21.15.26:0/2369666133 wait complete. 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f6925477700 1 Processor -- start 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f6925477700 1 -- start start 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f6925477700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f6925477700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f692010cac0 con 0x7f6920108560 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f691effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f691effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55078/0 (socket says 172.21.15.26:55078) 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.053+0000 7f691effd700 1 -- 172.21.15.26:0/672171567 learned_addr learned my addr 172.21.15.26:0/672171567 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:22.720 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f691effd700 1 -- 172.21.15.26:0/672171567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692019ca00 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f691effd700 1 --2- 172.21.15.26:0/672171567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f691800b7b0 tx=0x7f691800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6918020020 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6918012430 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6918017040 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692019cc90 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f692019f980 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6918023020 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.054+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f691800f050 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.055+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69200640a0 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.057+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f691802a020 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.129+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f692019fb50 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6917fff700 1 -- 172.21.15.26:0/672171567 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f691800b860 con 0x7f6920108560 2024-04-23T18:07:22.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 msgr2=0x7f692019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6925477700 1 --2- 172.21.15.26:0/672171567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f691800b7b0 tx=0x7f691800be20 comp rx=0 tx=0).stop 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 shutdown_connections 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6925477700 1 --2- 172.21.15.26:0/672171567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6920108560 0x7f692019c4c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.132+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 >> 172.21.15.26:0/672171567 conn(0x7f6920103b40 msgr2=0x7f69201046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.133+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 shutdown_connections 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:22.133+0000 7f6925477700 1 -- 172.21.15.26:0/672171567 wait complete. 2024-04-23T18:07:22.722 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T18:07:22.742 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:07:22.743 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mgr... 2024-04-23T18:07:22.743 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T18:07:22.743 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T18:07:23.000 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:07:23.000 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-23T18:07:23.220 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968.target.wants/ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-2a701120-019c-11ef-bc93-c7b262605968@.service. 2024-04-23T18:07:23.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:23 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/672171567' entity='client.admin' 2024-04-23T18:07:23.591 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:23 smithi026 podman[29579]: 2024-04-23 18:07:23.36917745 +0000 UTC m=+0.017876440 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:23.854 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:23 smithi026 podman[29579]: 2024-04-23 18:07:23.743321217 +0000 UTC m=+0.392020197 container create 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T18:07:24.028 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:07:24.028 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-23T18:07:24.038 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:07:24.039 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:07:24.039 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-23T18:07:24.039 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:07:24.044 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:07:24.044 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-23T18:07:24.052 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:07:24.052 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:07:24.052 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-23T18:07:24.052 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T18:07:24.053 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr to start... 2024-04-23T18:07:24.053 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr... 2024-04-23T18:07:24.207 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:23 smithi026 podman[29579]: 2024-04-23 18:07:23.937245863 +0000 UTC m=+0.585944844 container init 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T18:07:24.207 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:23 smithi026 podman[29579]: 2024-04-23 18:07:23.939978292 +0000 UTC m=+0.588677274 container start 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:07:24.207 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 bash[29579]: 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 2024-04-23T18:07:24.207 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 systemd[1]: Started Ceph mgr.x for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:07:24.207 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:24.106+0000 7fc740321200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:07:24.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:24.300+0000 7fc740321200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:24.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:24.408+0000 7fc740321200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:07:24.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2630771948' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:07:24.827 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:24.743+0000 7fc740321200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:07:24.827 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:24.832+0000 7fc740321200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:07:25.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:25.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:25.257 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:07:25.258 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:07:25.259 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:07:16.482472+0000", 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:25.260 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:07:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.551+0000 7f1ba97bc700 1 Processor -- start 2024-04-23T18:07:25.261 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.551+0000 7f1ba97bc700 1 -- start start 2024-04-23T18:07:25.265 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.551+0000 7f1ba97bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.551+0000 7f1ba97bc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba407bde0 con 0x7f1ba41090a0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55100/0 (socket says 172.21.15.26:55100) 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba2ffd700 1 -- 172.21.15.26:0/2553803646 learned_addr learned my addr 172.21.15.26:0/2553803646 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba2ffd700 1 -- 172.21.15.26:0/2553803646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba410b9d0 con 0x7f1ba41090a0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba2ffd700 1 --2- 172.21.15.26:0/2553803646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098a0 tx=0x7f1b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd84e2d68880c937 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba1ffb700 1 -- 172.21.15.26:0/2553803646 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b98004160 con 0x7f1ba41090a0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba1ffb700 1 -- 172.21.15.26:0/2553803646 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b98004300 con 0x7f1ba41090a0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.552+0000 7f1ba1ffb700 1 -- 172.21.15.26:0/2553803646 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b98011af0 con 0x7f1ba41090a0 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2553803646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 msgr2=0x7f1ba410b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 --2- 172.21.15.26:0/2553803646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098a0 tx=0x7f1b98009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2553803646 shutdown_connections 2024-04-23T18:07:25.266 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 --2- 172.21.15.26:0/2553803646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba410b490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2553803646 >> 172.21.15.26:0/2553803646 conn(0x7f1ba4102a50 msgr2=0x7f1ba4104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2553803646 shutdown_connections 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2553803646 wait complete. 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.553+0000 7f1ba97bc700 1 Processor -- start 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba97bc700 1 -- start start 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba97bc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba97bc700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba410c8d0 con 0x7f1ba41090a0 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55114/0 (socket says 172.21.15.26:55114) 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba2ffd700 1 -- 172.21.15.26:0/2630771948 learned_addr learned my addr 172.21.15.26:0/2630771948 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba2ffd700 1 -- 172.21.15.26:0/2630771948 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba419ec20 con 0x7f1ba41090a0 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1ba2ffd700 1 --2- 172.21.15.26:0/2630771948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b98011ac0 tx=0x7f1b98011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b98003970 con 0x7f1ba41090a0 2024-04-23T18:07:25.267 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.554+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b98003b10 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.555+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba419eeb0 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.555+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b98024610 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.555+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba41a1ba0 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.555+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1b98024c20 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.555+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b98019700 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.556+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba40640a0 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.560+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1b98043e50 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.635+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1ba41a1d70 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.635+0000 7f1b8bfff700 1 -- 172.21.15.26:0/2630771948 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1b98032030 con 0x7f1ba41090a0 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 msgr2=0x7f1ba419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 --2- 172.21.15.26:0/2630771948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b98011ac0 tx=0x7f1b98011ba0 comp rx=0 tx=0).stop 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 shutdown_connections 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 --2- 172.21.15.26:0/2630771948 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1ba41090a0 0x7f1ba419e6e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:25.268 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 >> 172.21.15.26:0/2630771948 conn(0x7f1ba4102a50 msgr2=0x7f1ba4103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:25.269 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 shutdown_connections 2024-04-23T18:07:25.269 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:24.636+0000 7f1ba97bc700 1 -- 172.21.15.26:0/2630771948 wait complete. 2024-04-23T18:07:25.269 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (1/15)... 2024-04-23T18:07:26.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:26.102+0000 7fc740321200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:07:26.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:26.282+0000 7fc740321200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:26.738 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:26.472+0000 7fc740321200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:07:27.089 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:26.779+0000 7fc740321200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:07:27.089 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:26.880+0000 7fc740321200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:07:27.089 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:27 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:27.094+0000 7fc740321200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:07:27.391 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:27 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:27.300+0000 7fc740321200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:27.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3980949367' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:07:28.343 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:27 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:27.965+0000 7fc740321200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:07:28.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:07:28.358 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:07:28.359 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:07:16.482472+0000", 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:28.360 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.674+0000 7fcb83c5f700 1 Processor -- start 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb83c5f700 1 -- start start 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb83c5f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb83c5f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c108ee0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb819fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb819fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55122/0 (socket says 172.21.15.26:55122) 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb819fb700 1 -- 172.21.15.26:0/2932404566 learned_addr learned my addr 172.21.15.26:0/2932404566 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.675+0000 7fcb819fb700 1 -- 172.21.15.26:0/2932404566 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c1096f0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb819fb700 1 --2- 172.21.15.26:0/2932404566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098a0 tx=0x7fcb78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad7355cc28722ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb809f9700 1 -- 172.21.15.26:0/2932404566 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb78004160 con 0x7fcb7c1084f0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb809f9700 1 -- 172.21.15.26:0/2932404566 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcb78004300 con 0x7fcb7c1084f0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb809f9700 1 -- 172.21.15.26:0/2932404566 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb78011af0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb83c5f700 1 -- 172.21.15.26:0/2932404566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 msgr2=0x7fcb7c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.676+0000 7fcb83c5f700 1 --2- 172.21.15.26:0/2932404566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098a0 tx=0x7fcb78009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:28.361 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 -- 172.21.15.26:0/2932404566 shutdown_connections 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 --2- 172.21.15.26:0/2932404566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c108910 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 -- 172.21.15.26:0/2932404566 >> 172.21.15.26:0/2932404566 conn(0x7fcb7c103b10 msgr2=0x7fcb7c105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 -- 172.21.15.26:0/2932404566 shutdown_connections 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 -- 172.21.15.26:0/2932404566 wait complete. 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 Processor -- start 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.677+0000 7fcb83c5f700 1 -- start start 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb83c5f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb83c5f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c10caa0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb819fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb819fb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55124/0 (socket says 172.21.15.26:55124) 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb819fb700 1 -- 172.21.15.26:0/3980949367 learned_addr learned my addr 172.21.15.26:0/3980949367 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb819fb700 1 -- 172.21.15.26:0/3980949367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c07cfb0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb819fb700 1 --2- 172.21.15.26:0/3980949367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb78011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:28.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb78003b20 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcb7801a8c0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c07b1a0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb78024940 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.678+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c07b6e0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.679+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcb78022020 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.679+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcb7802ddd0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.680+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c0640a0 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.684+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcb78042e50 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.771+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcb7c07b990 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.771+0000 7fcb72ffd700 1 -- 172.21.15.26:0/3980949367 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcb78031020 con 0x7fcb7c1084f0 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 msgr2=0x7fcb7c07ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 --2- 172.21.15.26:0/3980949367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb78011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 shutdown_connections 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 --2- 172.21.15.26:0/3980949367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb7c1084f0 0x7fcb7c07ca70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 >> 172.21.15.26:0/3980949367 conn(0x7fcb7c103b10 msgr2=0x7fcb7c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:28.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 shutdown_connections 2024-04-23T18:07:28.364 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:27.772+0000 7fcb83c5f700 1 -- 172.21.15.26:0/3980949367 wait complete. 2024-04-23T18:07:28.364 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (2/15)... 2024-04-23T18:07:28.709 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:28 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:28.407+0000 7fc740321200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:07:28.710 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:28 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:28.508+0000 7fc740321200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:07:29.209 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:28 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:28.717+0000 7fc740321200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:07:29.617 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.273+0000 7fc740321200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:07:29.617 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.368+0000 7fc740321200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:07:29.960 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.622+0000 7fc740321200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:07:29.960 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.760+0000 7fc740321200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:07:29.960 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.859+0000 7fc740321200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:07:30.286 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:29.979+0000 7fc740321200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:30.286 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:30 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:30.291+0000 7fc740321200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:07:31.183 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:30 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/134952140' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:07:31.183 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:30 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:30.958+0000 7fc740321200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:07:31.183 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:31.188+0000 7fc740321200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.479 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:31.480 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:07:31.481 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:07:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:07:31.490 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:07:16.482472+0000", 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.777+0000 7efc6e440700 1 Processor -- start 2024-04-23T18:07:31.491 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.777+0000 7efc6e440700 1 -- start start 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.777+0000 7efc6e440700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.777+0000 7efc6e440700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7efc68107e30 con 0x7efc68107440 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.777+0000 7efc67fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc67fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52088/0 (socket says 172.21.15.26:52088) 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc67fff700 1 -- 172.21.15.26:0/1239152646 learned_addr learned my addr 172.21.15.26:0/1239152646 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc67fff700 1 -- 172.21.15.26:0/1239152646 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc68108640 con 0x7efc68107440 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc67fff700 1 --2- 172.21.15.26:0/1239152646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efc500098a0 tx=0x7efc50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c00090a9cbcbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc66ffd700 1 -- 172.21.15.26:0/1239152646 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc50004160 con 0x7efc68107440 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc66ffd700 1 -- 172.21.15.26:0/1239152646 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efc50004300 con 0x7efc68107440 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.778+0000 7efc66ffd700 1 -- 172.21.15.26:0/1239152646 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc50011af0 con 0x7efc68107440 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 -- 172.21.15.26:0/1239152646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 msgr2=0x7efc68107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 --2- 172.21.15.26:0/1239152646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efc500098a0 tx=0x7efc50009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:31.492 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 -- 172.21.15.26:0/1239152646 shutdown_connections 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 --2- 172.21.15.26:0/1239152646 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc68107860 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 -- 172.21.15.26:0/1239152646 >> 172.21.15.26:0/1239152646 conn(0x7efc68102a40 msgr2=0x7efc68104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 -- 172.21.15.26:0/1239152646 shutdown_connections 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 -- 172.21.15.26:0/1239152646 wait complete. 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.779+0000 7efc6e440700 1 Processor -- start 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc6e440700 1 -- start start 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc6e440700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc6e440700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7efc6810b9f0 con 0x7efc68107440 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc67fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc67fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52096/0 (socket says 172.21.15.26:52096) 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc67fff700 1 -- 172.21.15.26:0/134952140 learned_addr learned my addr 172.21.15.26:0/134952140 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc67fff700 1 -- 172.21.15.26:0/134952140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6819ca60 con 0x7efc68107440 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.780+0000 7efc67fff700 1 --2- 172.21.15.26:0/134952140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efc50000c00 tx=0x7efc50011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc50003b20 con 0x7efc68107440 2024-04-23T18:07:31.493 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efc5001a8c0 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc50024940 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6819ccf0 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6819f990 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efc50022020 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.781+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efc500195c0 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.782+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc680640a0 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.784+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efc50042e50 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.867+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efc6802e190 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.868+0000 7efc657fa700 1 -- 172.21.15.26:0/134952140 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efc50031030 con 0x7efc68107440 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 msgr2=0x7efc6819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 --2- 172.21.15.26:0/134952140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efc50000c00 tx=0x7efc50011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 shutdown_connections 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 --2- 172.21.15.26:0/134952140 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efc68107440 0x7efc6819c520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:31.494 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 >> 172.21.15.26:0/134952140 conn(0x7efc68102a40 msgr2=0x7efc681038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:31.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 shutdown_connections 2024-04-23T18:07:31.495 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:30.869+0000 7efc6e440700 1 -- 172.21.15.26:0/134952140 wait complete. 2024-04-23T18:07:31.495 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (3/15)... 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: Activating manager daemon x 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: mgrmap e2: x(active, starting, since 0.00324448s) 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:07:32.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: Manager daemon x is now available 2024-04-23T18:07:32.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:07:32.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:07:32.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' 2024-04-23T18:07:32.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' 2024-04-23T18:07:32.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:31 smithi026 ceph-mon[29364]: from='mgr.14100 172.21.15.26:0/2819112697' entity='mgr.x' 2024-04-23T18:07:33.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:33 smithi026 ceph-mon[29364]: mgrmap e3: x(active, since 1.00703s) 2024-04-23T18:07:34.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:34 smithi026 ceph-mon[29364]: mgrmap e4: x(active, since 2s) 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:07:34.809 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:07:34.810 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:07:34.819 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:07:34.819 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:07:34.819 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:07:34.820 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.821 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:07:16.482472+0000", 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac8b32700 1 Processor -- start 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac8b32700 1 -- start start 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac8b32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac8b32700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffac4107060 con 0x7ffac4108b20 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52158/0 (socket says 172.21.15.26:52158) 2024-04-23T18:07:34.822 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.918+0000 7ffac259c700 1 -- 172.21.15.26:0/1660251185 learned_addr learned my addr 172.21.15.26:0/1660251185 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.919+0000 7ffac259c700 1 -- 172.21.15.26:0/1660251185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac41071e0 con 0x7ffac4108b20 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.919+0000 7ffac259c700 1 --2- 172.21.15.26:0/1660251185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffab40098a0 tx=0x7ffab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6816ad6fb17af8b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.919+0000 7ffac159a700 1 -- 172.21.15.26:0/1660251185 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab4004160 con 0x7ffac4108b20 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.919+0000 7ffac159a700 1 -- 172.21.15.26:0/1660251185 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffab4004300 con 0x7ffac4108b20 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.919+0000 7ffac159a700 1 -- 172.21.15.26:0/1660251185 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab4011af0 con 0x7ffac4108b20 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 -- 172.21.15.26:0/1660251185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 msgr2=0x7ffac4108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 --2- 172.21.15.26:0/1660251185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffab40098a0 tx=0x7ffab4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 -- 172.21.15.26:0/1660251185 shutdown_connections 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 --2- 172.21.15.26:0/1660251185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4108f40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 -- 172.21.15.26:0/1660251185 >> 172.21.15.26:0/1660251185 conn(0x7ffac4102a60 msgr2=0x7ffac4104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 -- 172.21.15.26:0/1660251185 shutdown_connections 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.920+0000 7ffac8b32700 1 -- 172.21.15.26:0/1660251185 wait complete. 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac8b32700 1 Processor -- start 2024-04-23T18:07:34.823 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac8b32700 1 -- start start 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac8b32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac8b32700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffac410b9f0 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52160/0 (socket says 172.21.15.26:52160) 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac259c700 1 -- 172.21.15.26:0/595654985 learned_addr learned my addr 172.21.15.26:0/595654985 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac259c700 1 -- 172.21.15.26:0/595654985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac4192b60 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffac259c700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009fb0 tx=0x7ffab4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.921+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab40036a0 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.922+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac4190da0 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.922+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffab401a430 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.922+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab4019550 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.922+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac4191280 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.922+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ffab4003840 con 0x7ffac4108b20 2024-04-23T18:07:34.824 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.923+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac410bb70 con 0x7ffac4108b20 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.923+0000 7ffab37fe700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 0x7ffaac03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.923+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffab404d880 con 0x7ffac4108b20 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.923+0000 7ffac1d9b700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 0x7ffaac03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.923+0000 7ffac1d9b700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 0x7ffaac03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffab8006fd0 tx=0x7ffab8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:33.928+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffab404d2a0 con 0x7ffac4108b20 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.221+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ffac41962b0 con 0x7ffac4108b20 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.222+0000 7ffab37fe700 1 -- 172.21.15.26:0/595654985 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ffab4003b30 con 0x7ffac4108b20 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 msgr2=0x7ffaac03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 0x7ffaac03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffab8006fd0 tx=0x7ffab8006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 msgr2=0x7ffac4192620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009fb0 tx=0x7ffab4011940 comp rx=0 tx=0).stop 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 shutdown_connections 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ffaac039510 0x7ffaac03b9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 --2- 172.21.15.26:0/595654985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffac4108b20 0x7ffac4192620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:34.825 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 >> 172.21.15.26:0/595654985 conn(0x7ffac4102a60 msgr2=0x7ffac41039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:34.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 shutdown_connections 2024-04-23T18:07:34.826 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:34.224+0000 7ffac8b32700 1 -- 172.21.15.26:0/595654985 wait complete. 2024-04-23T18:07:34.826 INFO:teuthology.orchestra.run.smithi026.stdout:mgr is available 2024-04-23T18:07:35.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:35 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/595654985' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:07:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.269+0000 7ff6ff5c5700 1 Processor -- start 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.269+0000 7ff6ff5c5700 1 -- start start 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.269+0000 7ff6ff5c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.269+0000 7ff6ff5c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f810b8e0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6fd361700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6fd361700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52166/0 (socket says 172.21.15.26:52166) 2024-04-23T18:07:36.090 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6fd361700 1 -- 172.21.15.26:0/3431209360 learned_addr learned my addr 172.21.15.26:0/3431209360 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6fd361700 1 -- 172.21.15.26:0/3431209360 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f810ba60 con 0x7ff6f8108fb0 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6fd361700 1 --2- 172.21.15.26:0/3431209360 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80098a0 tx=0x7ff6e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373b690b5defc76 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6effff700 1 -- 172.21.15.26:0/3431209360 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e8004160 con 0x7ff6f8108fb0 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6effff700 1 -- 172.21.15.26:0/3431209360 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff6e8004300 con 0x7ff6f8108fb0 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.270+0000 7ff6effff700 1 -- 172.21.15.26:0/3431209360 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e8011af0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3431209360 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 msgr2=0x7ff6f810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3431209360 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80098a0 tx=0x7ff6e8009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3431209360 shutdown_connections 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3431209360 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f810b3a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3431209360 >> 172.21.15.26:0/3431209360 conn(0x7ff6f8102a60 msgr2=0x7ff6f8104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3431209360 shutdown_connections 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.271+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3431209360 wait complete. 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6ff5c5700 1 Processor -- start 2024-04-23T18:07:36.091 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6ff5c5700 1 -- start start 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6ff5c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6ff5c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f810c9c0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6fd361700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6fd361700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52170/0 (socket says 172.21.15.26:52170) 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6fd361700 1 -- 172.21.15.26:0/3633921375 learned_addr learned my addr 172.21.15.26:0/3633921375 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6fd361700 1 -- 172.21.15.26:0/3633921375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f819cb30 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.272+0000 7ff6fd361700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009fb0 tx=0x7ff6e8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e80036a0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff6e801a430 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f819cdc0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e8019550 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f819fab0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.273+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff6e8019850 con 0x7ff6f8108fb0 2024-04-23T18:07:36.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.274+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f80640a0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.274+0000 7ff6ee7fc700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 0x7ff6e403b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.274+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff6e804da10 con 0x7ff6f8108fb0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.274+0000 7ff6fcb60700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 0x7ff6e403b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.274+0000 7ff6fcb60700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 0x7ff6e403b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006fd0 tx=0x7ff6f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.279+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6e804d430 con 0x7ff6f8108fb0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.487+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff6f81a1170 con 0x7ff6f8108fb0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.488+0000 7ff6ee7fc700 1 -- 172.21.15.26:0/3633921375 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7ff6e801a5e0 con 0x7ff6f8108fb0 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 msgr2=0x7ff6e403b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 0x7ff6e403b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006fd0 tx=0x7ff6f4006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 msgr2=0x7ff6f819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009fb0 tx=0x7ff6e8011940 comp rx=0 tx=0).stop 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 shutdown_connections 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7ff6e4039510 0x7ff6e403b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 --2- 172.21.15.26:0/3633921375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff6f8108fb0 0x7ff6f819c5f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.490+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 >> 172.21.15.26:0/3633921375 conn(0x7ff6f8102a60 msgr2=0x7ff6f8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:36.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.491+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 shutdown_connections 2024-04-23T18:07:36.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:35.491+0000 7ff6ff5c5700 1 -- 172.21.15.26:0/3633921375 wait complete. 2024-04-23T18:07:36.094 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling cephadm module... 2024-04-23T18:07:36.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:36 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3633921375' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:07:37.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2223650420' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T18:07:37.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setuser ceph since I am not root 2024-04-23T18:07:37.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setgroup ceph since I am not root 2024-04-23T18:07:37.460 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:37.375+0000 7f69d6aa4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:07:37.783 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:37.568+0000 7f69d6aa4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:37.783 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:37.674+0000 7f69d6aa4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:07:37.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.506+0000 7fbe3b6e9700 1 Processor -- start 2024-04-23T18:07:37.791 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe3b6e9700 1 -- start start 2024-04-23T18:07:37.791 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe3b6e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe3b6e9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3407c8e0 con 0x7fbe3407de60 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe39485700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe39485700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52184/0 (socket says 172.21.15.26:52184) 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe39485700 1 -- 172.21.15.26:0/3344212423 learned_addr learned my addr 172.21.15.26:0/3344212423 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.507+0000 7fbe39485700 1 -- 172.21.15.26:0/3344212423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3407ca60 con 0x7fbe3407de60 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe39485700 1 --2- 172.21.15.26:0/3344212423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098a0 tx=0x7fbe30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d5a03b22c0d3cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe2bfff700 1 -- 172.21.15.26:0/3344212423 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30004160 con 0x7fbe3407de60 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe2bfff700 1 -- 172.21.15.26:0/3344212423 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbe30004300 con 0x7fbe3407de60 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe2bfff700 1 -- 172.21.15.26:0/3344212423 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30011af0 con 0x7fbe3407de60 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/3344212423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 msgr2=0x7fbe3407c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/3344212423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098a0 tx=0x7fbe30009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/3344212423 shutdown_connections 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/3344212423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3407c310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:37.792 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/3344212423 >> 172.21.15.26:0/3344212423 conn(0x7fbe34103b40 msgr2=0x7fbe34105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.508+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/3344212423 shutdown_connections 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/3344212423 wait complete. 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe3b6e9700 1 Processor -- start 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe3b6e9700 1 -- start start 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe3b6e9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe3b6e9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410eab0 con 0x7fbe3407de60 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.509+0000 7fbe39485700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe39485700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52186/0 (socket says 172.21.15.26:52186) 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe39485700 1 -- 172.21.15.26:0/2223650420 learned_addr learned my addr 172.21.15.26:0/2223650420 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe39485700 1 -- 172.21.15.26:0/2223650420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3419ebe0 con 0x7fbe3407de60 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe39485700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009fb0 tx=0x7fbe30011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe300036a0 con 0x7fbe3407de60 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbe3001a430 con 0x7fbe3407de60 2024-04-23T18:07:37.793 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30019550 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3419ee70 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.510+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe341a1b60 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.511+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fbe30003840 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.511+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe340640a0 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.512+0000 7fbe2a7fc700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 0x7fbe2003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.512+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbe3004d830 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.512+0000 7fbe38c84700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 0x7fbe2003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.512+0000 7fbe38c84700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 0x7fbe2003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe24006fd0 tx=0x7fbe24006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.517+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe30023e00 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:36.784+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fbe341a1d30 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.201+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fbe341a1d30 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.203+0000 7fbe2a7fc700 1 -- 172.21.15.26:0/2223650420 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fbe30023390 con 0x7fbe3407de60 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 msgr2=0x7fbe2003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:37.794 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 0x7fbe2003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe24006fd0 tx=0x7fbe24006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 msgr2=0x7fbe3419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009fb0 tx=0x7fbe30011940 comp rx=0 tx=0).stop 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 shutdown_connections 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7fbe20039560 0x7fbe2003ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 --2- 172.21.15.26:0/2223650420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbe3407de60 0x7fbe3419e6a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 >> 172.21.15.26:0/2223650420 conn(0x7fbe34103b40 msgr2=0x7fbe341046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 shutdown_connections 2024-04-23T18:07:37.795 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:37.204+0000 7fbe3b6e9700 1 -- 172.21.15.26:0/2223650420 wait complete. 2024-04-23T18:07:38.097 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:38.011+0000 7f69d6aa4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:07:38.097 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:38.101+0000 7f69d6aa4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:07:38.444 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:38 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2223650420' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T18:07:38.444 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:38 smithi026 ceph-mon[29364]: mgrmap e5: x(active, since 6s) 2024-04-23T18:07:39.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:39.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T18:07:39.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:07:39.191 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.243+0000 7f5db2499700 1 Processor -- start 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.243+0000 7f5db2499700 1 -- start start 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.243+0000 7f5db2499700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.243+0000 7f5db2499700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac107e30 con 0x7f5dac107440 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.243+0000 7f5dabfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5dabfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48044/0 (socket says 172.21.15.26:48044) 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5dabfff700 1 -- 172.21.15.26:0/4136864798 learned_addr learned my addr 172.21.15.26:0/4136864798 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5dabfff700 1 -- 172.21.15.26:0/4136864798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dac108640 con 0x7f5dac107440 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5dabfff700 1 --2- 172.21.15.26:0/4136864798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5da00098a0 tx=0x7f5da0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d988862abe22390e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5daaffd700 1 -- 172.21.15.26:0/4136864798 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5da0004160 con 0x7f5dac107440 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5daaffd700 1 -- 172.21.15.26:0/4136864798 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5da0004300 con 0x7f5dac107440 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.244+0000 7f5daaffd700 1 -- 172.21.15.26:0/4136864798 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5da0011af0 con 0x7f5dac107440 2024-04-23T18:07:39.192 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 -- 172.21.15.26:0/4136864798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 msgr2=0x7f5dac107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 --2- 172.21.15.26:0/4136864798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5da00098a0 tx=0x7f5da0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 -- 172.21.15.26:0/4136864798 shutdown_connections 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 --2- 172.21.15.26:0/4136864798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac107860 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 -- 172.21.15.26:0/4136864798 >> 172.21.15.26:0/4136864798 conn(0x7f5dac102a40 msgr2=0x7f5dac104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 -- 172.21.15.26:0/4136864798 shutdown_connections 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.245+0000 7f5db2499700 1 -- 172.21.15.26:0/4136864798 wait complete. 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5db2499700 1 Processor -- start 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5db2499700 1 -- start start 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5db2499700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5db2499700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac10b9f0 con 0x7f5dac107440 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5dabfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5dabfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48054/0 (socket says 172.21.15.26:48054) 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.246+0000 7f5dabfff700 1 -- 172.21.15.26:0/30124536 learned_addr learned my addr 172.21.15.26:0/30124536 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:39.193 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5dabfff700 1 -- 172.21.15.26:0/30124536 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dac19f220 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5dabfff700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5da0003b20 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5da001a8c0 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5da0024940 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dac19f3f0 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.247+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dac19f790 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5da0022020 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5da97fa700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 0x7f5d9403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5da004d4a0 con 0x7f5dac107440 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5dab7fe700 1 -- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 msgr2=0x7f5d9403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5dab7fe700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 0x7f5d9403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:07:39.194 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.248+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dac0640a0 con 0x7f5dac107440 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.254+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5da004d200 con 0x7f5dac107440 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.449+0000 7f5dab7fe700 1 -- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 msgr2=0x7f5d9403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.449+0000 7f5dab7fe700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 0x7f5d9403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.584+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5dac19ff10 con 0x7f5dac107440 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.585+0000 7f5da97fa700 1 -- 172.21.15.26:0/30124536 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f5da0021d80 con 0x7f5dac107440 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 msgr2=0x7f5d9403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 0x7f5d9403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 msgr2=0x7f5dac063900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5da0000c00 tx=0x7f5da0011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 shutdown_connections 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f5d94039600 0x7f5d9403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 --2- 172.21.15.26:0/30124536 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5dac107440 0x7f5dac063900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 >> 172.21.15.26:0/30124536 conn(0x7f5dac102a40 msgr2=0x7f5dac1035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 shutdown_connections 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:38.587+0000 7f5db2499700 1 -- 172.21.15.26:0/30124536 wait complete. 2024-04-23T18:07:39.195 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-04-23T18:07:39.196 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 5... 2024-04-23T18:07:39.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:39 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/30124536' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:07:39.568 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:39.389+0000 7f69d6aa4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:07:39.568 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:39.572+0000 7f69d6aa4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:39.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:39.749+0000 7f69d6aa4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:07:40.352 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:40.050+0000 7f69d6aa4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:07:40.352 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:40.146+0000 7f69d6aa4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:07:40.352 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:40.356+0000 7f69d6aa4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:07:40.709 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:40.557+0000 7f69d6aa4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:07:41.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:41.183+0000 7f69d6aa4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:07:41.927 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:41.622+0000 7f69d6aa4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:07:41.927 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:41.722+0000 7f69d6aa4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:07:41.927 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:41.931+0000 7f69d6aa4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:07:42.821 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:42.476+0000 7f69d6aa4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:07:42.821 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:42.572+0000 7f69d6aa4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:07:43.179 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:42.825+0000 7f69d6aa4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:07:43.179 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:42.964+0000 7f69d6aa4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:07:43.179 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:43.063+0000 7f69d6aa4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:07:43.179 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:43.183+0000 7f69d6aa4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:07:43.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:43.494+0000 7f69d6aa4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:07:44.395 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:44.123+0000 7f69d6aa4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:07:44.395 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:07:44.347+0000 7f69d6aa4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:07:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: Active manager daemon x restarted 2024-04-23T18:07:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: Activating manager daemon x 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: mgrmap e6: x(active, starting, since 0.00358726s) 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:07:44.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: Manager daemon x is now available 2024-04-23T18:07:44.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:44.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:44.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:44.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:44.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:44 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: Found migration_current of "None". Setting to last migration. 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: [23/Apr/2024:18:07:45] ENGINE Bus STARTING 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: mgrmap e7: x(active, since 1.00612s) 2024-04-23T18:07:45.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:45 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.592+0000 7f651d69f700 1 Processor -- start 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.592+0000 7f651d69f700 1 -- start start 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f651d69f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f651d69f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6518107e30 con 0x7f6518107440 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6516ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6516ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48056/0 (socket says 172.21.15.26:48056) 2024-04-23T18:07:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6516ffd700 1 -- 172.21.15.26:0/3673350365 learned_addr learned my addr 172.21.15.26:0/3673350365 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6516ffd700 1 -- 172.21.15.26:0/3673350365 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6518108640 con 0x7f6518107440 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6516ffd700 1 --2- 172.21.15.26:0/3673350365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f650c0098a0 tx=0x7f650c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74154402eff4f12f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6515ffb700 1 -- 172.21.15.26:0/3673350365 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f650c004160 con 0x7f6518107440 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6515ffb700 1 -- 172.21.15.26:0/3673350365 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f650c004300 con 0x7f6518107440 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.593+0000 7f6515ffb700 1 -- 172.21.15.26:0/3673350365 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f650c011af0 con 0x7f6518107440 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 -- 172.21.15.26:0/3673350365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 msgr2=0x7f6518107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 --2- 172.21.15.26:0/3673350365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f650c0098a0 tx=0x7f650c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 -- 172.21.15.26:0/3673350365 shutdown_connections 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 --2- 172.21.15.26:0/3673350365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f6518107860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 -- 172.21.15.26:0/3673350365 >> 172.21.15.26:0/3673350365 conn(0x7f6518102a40 msgr2=0x7f6518104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 -- 172.21.15.26:0/3673350365 shutdown_connections 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.594+0000 7f651d69f700 1 -- 172.21.15.26:0/3673350365 wait complete. 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f651d69f700 1 Processor -- start 2024-04-23T18:07:45.941 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f651d69f700 1 -- start start 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f651d69f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f651d69f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f651810b9f0 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f6516ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f6516ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48068/0 (socket says 172.21.15.26:48068) 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.595+0000 7f6516ffd700 1 -- 172.21.15.26:0/2182970612 learned_addr learned my addr 172.21.15.26:0/2182970612 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f6516ffd700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651819ca20 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f6516ffd700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f650c000c00 tx=0x7f650c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f650c003b20 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f650c01a8c0 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651819ccb0 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f650c024940 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.596+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651819f9a0 con 0x7f6518107440 2024-04-23T18:07:45.942 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f650c022020 con 0x7f6518107440 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f64fffff700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6518069ba0 con 0x7f6500039600 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f650c01a430 con 0x7f6518107440 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f65167fc700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.597+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.798+0000 7f65167fc700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:39.798+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:40.199+0000 7f65167fc700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:40.199+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:41.000+0000 7f65167fc700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:41.000+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:07:45.943 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:42.601+0000 7f65167fc700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/259946325 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:42.601+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:44.351+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f650c02dc90 con 0x7f6518107440 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:44.351+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 msgr2=0x7f650003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:44.351+0000 7f64fffff700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.352+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f650c006ea0 con 0x7f6518107440 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.352+0000 7f64fffff700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 0x7f650003f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.352+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f650c021b60 con 0x7f650003cd00 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.352+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 0x7f650003f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.353+0000 7f65167fc700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 0x7f650003f0f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6508003a10 tx=0x7f65080092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.354+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f650c021b60 con 0x7f650003cd00 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6518075460 con 0x7f650003cd00 2024-04-23T18:07:45.944 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f64fffff700 1 -- 172.21.15.26:0/2182970612 <== mgr.14120 v2:172.21.15.26:6800/2629443403 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f6518075460 con 0x7f650003cd00 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 msgr2=0x7f650003f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f651d69f700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 0x7f650003f0f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6508003a10 tx=0x7f65080092b0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 msgr2=0x7f651819c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.361+0000 7f651d69f700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f650c000c00 tx=0x7f650c011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 shutdown_connections 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f650003cd00 0x7f650003f0f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6518107440 0x7f651819c4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 --2- 172.21.15.26:0/2182970612 >> [v2:172.21.15.26:6800/259946325,v1:172.21.15.26:6801/259946325] conn(0x7f6500039600 0x7f650003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 >> 172.21.15.26:0/2182970612 conn(0x7f6518102a40 msgr2=0x7f65181038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 shutdown_connections 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:45.362+0000 7f651d69f700 1 -- 172.21.15.26:0/2182970612 wait complete. 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 5 is available 2024-04-23T18:07:45.945 INFO:teuthology.orchestra.run.smithi026.stdout:Setting orchestrator backend to cephadm... 2024-04-23T18:07:46.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:46 smithi026 ceph-mon[29364]: [23/Apr/2024:18:07:45] ENGINE Serving on https://172.21.15.26:7150 2024-04-23T18:07:46.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:46 smithi026 ceph-mon[29364]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T18:07:46.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:46 smithi026 ceph-mon[29364]: [23/Apr/2024:18:07:45] ENGINE Serving on http://172.21.15.26:8765 2024-04-23T18:07:46.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:46 smithi026 ceph-mon[29364]: [23/Apr/2024:18:07:45] ENGINE Bus STARTED 2024-04-23T18:07:46.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:46 smithi026 ceph-mon[29364]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef9323700 1 Processor -- start 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef9323700 1 -- start start 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef9323700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef9323700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faef407bec0 con 0x7faef4108fb0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48146/0 (socket says 172.21.15.26:48146) 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.424+0000 7faef2ffd700 1 -- 172.21.15.26:0/3388043960 learned_addr learned my addr 172.21.15.26:0/3388043960 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef2ffd700 1 -- 172.21.15.26:0/3388043960 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef410b8e0 con 0x7faef4108fb0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef2ffd700 1 --2- 172.21.15.26:0/3388043960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faee40098a0 tx=0x7faee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4386e9da0eab72e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef1ffb700 1 -- 172.21.15.26:0/3388043960 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faee4004160 con 0x7faef4108fb0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef1ffb700 1 -- 172.21.15.26:0/3388043960 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faee4004300 con 0x7faef4108fb0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef1ffb700 1 -- 172.21.15.26:0/3388043960 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faee4011af0 con 0x7faef4108fb0 2024-04-23T18:07:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef9323700 1 -- 172.21.15.26:0/3388043960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 msgr2=0x7faef410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.425+0000 7faef9323700 1 --2- 172.21.15.26:0/3388043960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faee40098a0 tx=0x7faee4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 -- 172.21.15.26:0/3388043960 shutdown_connections 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 --2- 172.21.15.26:0/3388043960 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef410b3a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 -- 172.21.15.26:0/3388043960 >> 172.21.15.26:0/3388043960 conn(0x7faef4102a40 msgr2=0x7faef4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 -- 172.21.15.26:0/3388043960 shutdown_connections 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 -- 172.21.15.26:0/3388043960 wait complete. 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 Processor -- start 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.426+0000 7faef9323700 1 -- start start 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef9323700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef9323700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faef410c8b0 con 0x7faef4108fb0 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:47.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48148/0 (socket says 172.21.15.26:48148) 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef2ffd700 1 -- 172.21.15.26:0/4116835918 learned_addr learned my addr 172.21.15.26:0/4116835918 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef2ffd700 1 -- 172.21.15.26:0/4116835918 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef419c9f0 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef2ffd700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faee4009fb0 tx=0x7faee4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faee40036a0 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faee401a430 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef419cc80 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.427+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faee4019550 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.428+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef419f970 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.428+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7faee4019850 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.428+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef40640a0 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.428+0000 7faedbfff700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 0x7faedc03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.429+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faee404db90 con 0x7faef4108fb0 2024-04-23T18:07:47.255 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.429+0000 7faef27fc700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 0x7faedc03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.429+0000 7faef27fc700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 0x7faedc03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faee8006fd0 tx=0x7faee8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.434+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faee400f2e0 con 0x7faef4108fb0 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.660+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7faef4103d30 con 0x7faedc039560 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.664+0000 7faedbfff700 1 -- 172.21.15.26:0/4116835918 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faef4103d30 con 0x7faedc039560 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.666+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 msgr2=0x7faedc03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.666+0000 7faef9323700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 0x7faedc03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faee8006fd0 tx=0x7faee8006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.666+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 msgr2=0x7faef419c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.666+0000 7faef9323700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faee4009fb0 tx=0x7faee4011940 comp rx=0 tx=0).stop 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 shutdown_connections 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faedc039560 0x7faedc03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 --2- 172.21.15.26:0/4116835918 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faef4108fb0 0x7faef419c4b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 >> 172.21.15.26:0/4116835918 conn(0x7faef4102a40 msgr2=0x7faef4103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 shutdown_connections 2024-04-23T18:07:47.256 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:46.667+0000 7faef9323700 1 -- 172.21.15.26:0/4116835918 wait complete. 2024-04-23T18:07:47.502 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:47 smithi026 ceph-mon[29364]: mgrmap e8: x(active, since 2s) 2024-04-23T18:07:47.503 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:47 smithi026 ceph-mon[29364]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:47.503 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:47 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:47.503 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:47 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:48.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:48 smithi026 ceph-mon[29364]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.719+0000 7f6ee2400700 1 Processor -- start 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.719+0000 7f6ee2400700 1 -- start start 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.719+0000 7f6ee2400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.719+0000 7f6ee2400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc108f50 con 0x7f6edc108560 2024-04-23T18:07:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.719+0000 7f6edbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48158/0 (socket says 172.21.15.26:48158) 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edbfff700 1 -- 172.21.15.26:0/466205090 learned_addr learned my addr 172.21.15.26:0/466205090 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edbfff700 1 -- 172.21.15.26:0/466205090 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc109710 con 0x7f6edc108560 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edbfff700 1 --2- 172.21.15.26:0/466205090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0098a0 tx=0x7f6ecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21172e2eed55ca2e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edaffd700 1 -- 172.21.15.26:0/466205090 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ecc004160 con 0x7f6edc108560 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edaffd700 1 -- 172.21.15.26:0/466205090 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6ecc004300 con 0x7f6edc108560 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.720+0000 7f6edaffd700 1 -- 172.21.15.26:0/466205090 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ecc011af0 con 0x7f6edc108560 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 -- 172.21.15.26:0/466205090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 msgr2=0x7f6edc108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 --2- 172.21.15.26:0/466205090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0098a0 tx=0x7f6ecc009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 -- 172.21.15.26:0/466205090 shutdown_connections 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 --2- 172.21.15.26:0/466205090 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc108980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 -- 172.21.15.26:0/466205090 >> 172.21.15.26:0/466205090 conn(0x7f6edc103b40 msgr2=0x7f6edc105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 -- 172.21.15.26:0/466205090 shutdown_connections 2024-04-23T18:07:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.721+0000 7f6ee2400700 1 -- 172.21.15.26:0/466205090 wait complete. 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6ee2400700 1 Processor -- start 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6ee2400700 1 -- start start 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6ee2400700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6ee2400700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc10cac0 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6edbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6edbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48164/0 (socket says 172.21.15.26:48164) 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.722+0000 7f6edbfff700 1 -- 172.21.15.26:0/1720686835 learned_addr learned my addr 172.21.15.26:0/1720686835 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6edbfff700 1 -- 172.21.15.26:0/1720686835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc07e0c0 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6edbfff700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009fb0 tx=0x7f6ecc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ecc0036a0 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6ecc01a430 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ecc019550 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc07c2b0 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.723+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc07c7f0 con 0x7f6edc108560 2024-04-23T18:07:48.544 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.724+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6ecc019800 con 0x7f6edc108560 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.724+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6edc069ba0 con 0x7f6edc108560 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.724+0000 7f6ed97fa700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 0x7f6ec803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.724+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6ecc04d8b0 con 0x7f6edc108560 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.724+0000 7f6edb7fe700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 0x7f6ec803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.725+0000 7f6edb7fe700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 0x7f6ec803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.730+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ecc00fda0 con 0x7f6edc108560 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.937+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f6edc104e70 con 0x7f6ec80395b0 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.938+0000 7f6ed97fa700 1 -- 172.21.15.26:0/1720686835 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f6edc104e70 con 0x7f6ec80395b0 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 msgr2=0x7f6ec803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 0x7f6ec803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 msgr2=0x7f6edc07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009fb0 tx=0x7f6ecc011940 comp rx=0 tx=0).stop 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 shutdown_connections 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f6ec80395b0 0x7f6ec803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:48.545 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 --2- 172.21.15.26:0/1720686835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6edc108560 0x7f6edc07db80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:48.546 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 >> 172.21.15.26:0/1720686835 conn(0x7f6edc103b40 msgr2=0x7f6edc104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:48.546 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 shutdown_connections 2024-04-23T18:07:48.546 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:47.941+0000 7f6ee2400700 1 -- 172.21.15.26:0/1720686835 wait complete. 2024-04-23T18:07:48.546 INFO:teuthology.orchestra.run.smithi026.stdout:Generating ssh key... 2024-04-23T18:07:49.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-mon[29364]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:49.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-mon[29364]: Generating ssh key... 2024-04-23T18:07:49.914 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: Generating public/private rsa key pair. 2024-04-23T18:07:49.956 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: Your identification has been saved in /tmp/tmpj2yggbqd/key. 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: Your public key has been saved in /tmp/tmpj2yggbqd/key.pub. 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: The key fingerprint is: 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: SHA256:kb8AuSdoatiSeTXxjhNYBFD055Qm4sycn2o3xzLCY9o ceph-2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: The key's randomart image is: 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: +---[RSA 3072]----+ 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: |.++.. | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | o o . | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | . = B o | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | = = X o o | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | B * = S . | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | = = * o . . | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: |=.= =.. . | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: | =*.=.o | 2024-04-23T18:07:49.957 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: |.+E+ = | 2024-04-23T18:07:49.958 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:07:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: +----[SHA256]-----+ 2024-04-23T18:07:50.207 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8bb959700 1 Processor -- start 2024-04-23T18:07:50.207 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8bb959700 1 -- start start 2024-04-23T18:07:50.207 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8bb959700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:50.207 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8bb959700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b4063d60 con 0x7fd8b4106da0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8b96f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8b96f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54800/0 (socket says 172.21.15.26:54800) 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8b96f5700 1 -- 172.21.15.26:0/1107080172 learned_addr learned my addr 172.21.15.26:0/1107080172 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8b96f5700 1 -- 172.21.15.26:0/1107080172 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b4107700 con 0x7fd8b4106da0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.959+0000 7fd8b96f5700 1 --2- 172.21.15.26:0/1107080172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098a0 tx=0x7fd8b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d8488dfda90ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8abfff700 1 -- 172.21.15.26:0/1107080172 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8b0004160 con 0x7fd8b4106da0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8abfff700 1 -- 172.21.15.26:0/1107080172 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd8b0004300 con 0x7fd8b4106da0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8abfff700 1 -- 172.21.15.26:0/1107080172 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8b0011af0 con 0x7fd8b4106da0 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 -- 172.21.15.26:0/1107080172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 msgr2=0x7fd8b41071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 --2- 172.21.15.26:0/1107080172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098a0 tx=0x7fd8b0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 -- 172.21.15.26:0/1107080172 shutdown_connections 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 --2- 172.21.15.26:0/1107080172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b41071c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:50.208 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 -- 172.21.15.26:0/1107080172 >> 172.21.15.26:0/1107080172 conn(0x7fd8b4102a40 msgr2=0x7fd8b4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 -- 172.21.15.26:0/1107080172 shutdown_connections 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.960+0000 7fd8bb959700 1 -- 172.21.15.26:0/1107080172 wait complete. 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8bb959700 1 Processor -- start 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8bb959700 1 -- start start 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8bb959700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8bb959700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b4108730 con 0x7fd8b4106da0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8b96f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8b96f5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54810/0 (socket says 172.21.15.26:54810) 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8b96f5700 1 -- 172.21.15.26:0/2704422596 learned_addr learned my addr 172.21.15.26:0/2704422596 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.961+0000 7fd8b96f5700 1 -- 172.21.15.26:0/2704422596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b419eba0 con 0x7fd8b4106da0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8b96f5700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0003f40 tx=0x7fd8b0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8b00038b0 con 0x7fd8b4106da0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd8b0003a50 con 0x7fd8b4106da0 2024-04-23T18:07:50.209 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b419ee30 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8b001a5c0 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.962+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b41a1b20 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd8b001a870 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8aa7fc700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 0x7fd8a003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd8b004da90 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8b8ef4700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 0x7fd8a003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b40640a0 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.963+0000 7fd8b8ef4700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 0x7fd8a003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006fd0 tx=0x7fd8a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:48.969+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8b0019c70 con 0x7fd8b4106da0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.179+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd8b402e2f0 con 0x7fd8a00395b0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.659+0000 7fd8aa7fc700 1 -- 172.21.15.26:0/2704422596 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8b402e2f0 con 0x7fd8a00395b0 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.661+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 msgr2=0x7fd8a003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:50.210 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.661+0000 7fd8bb959700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 0x7fd8a003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006fd0 tx=0x7fd8a4006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.661+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 msgr2=0x7fd8b419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.661+0000 7fd8bb959700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0003f40 tx=0x7fd8b0011890 comp rx=0 tx=0).stop 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 shutdown_connections 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7fd8a00395b0 0x7fd8a003ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 --2- 172.21.15.26:0/2704422596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd8b4106da0 0x7fd8b419e660 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 >> 172.21.15.26:0/2704422596 conn(0x7fd8b4102a40 msgr2=0x7fd8b4195300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 shutdown_connections 2024-04-23T18:07:50.211 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:49.662+0000 7fd8bb959700 1 -- 172.21.15.26:0/2704422596 wait complete. 2024-04-23T18:07:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:50 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:50 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:51.470 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.669+0000 7faf8976e700 1 Processor -- start 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.669+0000 7faf8976e700 1 -- start start 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.669+0000 7faf8976e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.669+0000 7faf8976e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faf8407c830 con 0x7faf8407de40 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54812/0 (socket says 172.21.15.26:54812) 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf82ffd700 1 -- 172.21.15.26:0/1034930318 learned_addr learned my addr 172.21.15.26:0/1034930318 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf82ffd700 1 -- 172.21.15.26:0/1034930318 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8407c9b0 con 0x7faf8407de40 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf82ffd700 1 --2- 172.21.15.26:0/1034930318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009e80 tx=0x7faf7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8afd00dfa4cac64 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf81ffb700 1 -- 172.21.15.26:0/1034930318 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf7c017070 con 0x7faf8407de40 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf81ffb700 1 -- 172.21.15.26:0/1034930318 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faf7c00b8c0 con 0x7faf8407de40 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.670+0000 7faf81ffb700 1 -- 172.21.15.26:0/1034930318 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf7c004490 con 0x7faf8407de40 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 -- 172.21.15.26:0/1034930318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 msgr2=0x7faf8407c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:51.471 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 --2- 172.21.15.26:0/1034930318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009e80 tx=0x7faf7c00b330 comp rx=0 tx=0).stop 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 -- 172.21.15.26:0/1034930318 shutdown_connections 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 --2- 172.21.15.26:0/1034930318 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8407c2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 -- 172.21.15.26:0/1034930318 >> 172.21.15.26:0/1034930318 conn(0x7faf84103b70 msgr2=0x7faf84105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 -- 172.21.15.26:0/1034930318 shutdown_connections 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.671+0000 7faf8976e700 1 -- 172.21.15.26:0/1034930318 wait complete. 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf8976e700 1 Processor -- start 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf8976e700 1 -- start start 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf8976e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf8976e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faf8410eae0 con 0x7faf8407de40 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf82ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54820/0 (socket says 172.21.15.26:54820) 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf82ffd700 1 -- 172.21.15.26:0/723226076 learned_addr learned my addr 172.21.15.26:0/723226076 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.672+0000 7faf82ffd700 1 -- 172.21.15.26:0/723226076 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8419eba0 con 0x7faf8407de40 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf82ffd700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00b7b0 tx=0x7faf7c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:51.472 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf7c020020 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faf7c012430 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8419ee30 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf7c017040 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.673+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf841a1b20 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.674+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7faf7c023020 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.674+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf840640a0 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.674+0000 7faf7bfff700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 0x7faf7003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.674+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faf7c00f050 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.674+0000 7faf827fc700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 0x7faf7003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.675+0000 7faf827fc700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 0x7faf7003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faf6c006fd0 tx=0x7faf6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.679+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf7c02a020 con 0x7faf8407de40 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.887+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7faf84104db0 con 0x7faf70039600 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.888+0000 7faf7bfff700 1 -- 172.21.15.26:0/723226076 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7faf84104db0 con 0x7faf70039600 2024-04-23T18:07:51.473 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 msgr2=0x7faf7003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 0x7faf7003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faf6c006fd0 tx=0x7faf6c006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 msgr2=0x7faf8419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00b7b0 tx=0x7faf7c00bda0 comp rx=0 tx=0).stop 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 shutdown_connections 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7faf70039600 0x7faf7003bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 --2- 172.21.15.26:0/723226076 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faf8407de40 0x7faf8419e660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 >> 172.21.15.26:0/723226076 conn(0x7faf84103b70 msgr2=0x7faf841046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 shutdown_connections 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:50.890+0000 7faf8976e700 1 -- 172.21.15.26:0/723226076 wait complete. 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T18:07:51.474 INFO:teuthology.orchestra.run.smithi026.stdout:Adding host smithi026... 2024-04-23T18:07:51.781 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:51 smithi026 ceph-mon[29364]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:52 smithi026 ceph-mon[29364]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:53 smithi026 ceph-mon[29364]: Deploying cephadm binary to smithi026 2024-04-23T18:07:56.970 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Added host 'smithi026' with addr '172.21.15.26' 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f885423b700 1 Processor -- start 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f885423b700 1 -- start start 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f885423b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f885423b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f884c07c830 con 0x7f884c07de40 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f8851fd7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f8851fd7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54834/0 (socket says 172.21.15.26:54834) 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f8851fd7700 1 -- 172.21.15.26:0/456281556 learned_addr learned my addr 172.21.15.26:0/456281556 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.898+0000 7f8851fd7700 1 -- 172.21.15.26:0/456281556 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c07c9b0 con 0x7f884c07de40 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f8851fd7700 1 --2- 172.21.15.26:0/456281556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88480098a0 tx=0x7f8848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b76aec04bd6a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f8850fd5700 1 -- 172.21.15.26:0/456281556 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8848004160 con 0x7f884c07de40 2024-04-23T18:07:56.971 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f8850fd5700 1 -- 172.21.15.26:0/456281556 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8848004300 con 0x7f884c07de40 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f8850fd5700 1 -- 172.21.15.26:0/456281556 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8848011af0 con 0x7f884c07de40 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f885423b700 1 -- 172.21.15.26:0/456281556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 msgr2=0x7f884c07c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f885423b700 1 --2- 172.21.15.26:0/456281556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88480098a0 tx=0x7f8848009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f885423b700 1 -- 172.21.15.26:0/456281556 shutdown_connections 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f885423b700 1 --2- 172.21.15.26:0/456281556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c07c2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.899+0000 7f885423b700 1 -- 172.21.15.26:0/456281556 >> 172.21.15.26:0/456281556 conn(0x7f884c103b40 msgr2=0x7f884c105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 -- 172.21.15.26:0/456281556 shutdown_connections 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 -- 172.21.15.26:0/456281556 wait complete. 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 Processor -- start 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 -- start start 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.900+0000 7f885423b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f884c10ead0 con 0x7f884c07de40 2024-04-23T18:07:56.972 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8851fd7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8851fd7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54836/0 (socket says 172.21.15.26:54836) 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8851fd7700 1 -- 172.21.15.26:0/4027911730 learned_addr learned my addr 172.21.15.26:0/4027911730 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8851fd7700 1 -- 172.21.15.26:0/4027911730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c19ec10 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8851fd7700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8848011ac0 tx=0x7f8848011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8848003970 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8848003b10 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884c19eea0 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8848024610 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.901+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884c1a1b90 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.902+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f884801a430 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.902+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f884c0640a0 con 0x7f884c07de40 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.902+0000 7f8842ffd700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 0x7f883803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:56.973 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.902+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f884804d910 con 0x7f884c07de40 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.902+0000 7f88517d6700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 0x7f883803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.903+0000 7f88517d6700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 0x7f883803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f883c006fd0 tx=0x7f883c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:51.907+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8848016260 con 0x7f884c07de40 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:52.125+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}) -- 0x7f884c104dd0 con 0x7f88380395b0 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.325+0000 7f8842ffd700 1 -- 172.21.15.26:0/4027911730 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f884c104dd0 con 0x7f88380395b0 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 msgr2=0x7f883803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 0x7f883803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f883c006fd0 tx=0x7f883c006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 msgr2=0x7f884c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8848011ac0 tx=0x7f8848011ba0 comp rx=0 tx=0).stop 2024-04-23T18:07:56.974 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 shutdown_connections 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f88380395b0 0x7f883803ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 --2- 172.21.15.26:0/4027911730 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f884c07de40 0x7f884c19e6d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 >> 172.21.15.26:0/4027911730 conn(0x7f884c103b40 msgr2=0x7f884c1046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 shutdown_connections 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:56.327+0000 7f885423b700 1 -- 172.21.15.26:0/4027911730 wait complete. 2024-04-23T18:07:56.975 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mon service... 2024-04-23T18:07:57.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:57 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:57.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:57 smithi026 ceph-mon[29364]: Added host smithi026 2024-04-23T18:07:57.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:57 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:07:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T18:07:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.391+0000 7f846250c700 1 Processor -- start 2024-04-23T18:07:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.391+0000 7f846250c700 1 -- start start 2024-04-23T18:07:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.391+0000 7f846250c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.391+0000 7f846250c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f845c107060 con 0x7f845c108b20 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.391+0000 7f845bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54848/0 (socket says 172.21.15.26:54848) 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845bfff700 1 -- 172.21.15.26:0/1322212826 learned_addr learned my addr 172.21.15.26:0/1322212826 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845bfff700 1 -- 172.21.15.26:0/1322212826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845c1071e0 con 0x7f845c108b20 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845bfff700 1 --2- 172.21.15.26:0/1322212826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f844c0098a0 tx=0x7f844c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fa8accf979978 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845affd700 1 -- 172.21.15.26:0/1322212826 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c004160 con 0x7f845c108b20 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845affd700 1 -- 172.21.15.26:0/1322212826 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f844c004300 con 0x7f845c108b20 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.392+0000 7f845affd700 1 -- 172.21.15.26:0/1322212826 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c011af0 con 0x7f845c108b20 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- 172.21.15.26:0/1322212826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 msgr2=0x7f845c108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 --2- 172.21.15.26:0/1322212826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f844c0098a0 tx=0x7f844c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:07:58.272 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- 172.21.15.26:0/1322212826 shutdown_connections 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 --2- 172.21.15.26:0/1322212826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c108f40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- 172.21.15.26:0/1322212826 >> 172.21.15.26:0/1322212826 conn(0x7f845c102a60 msgr2=0x7f845c104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- 172.21.15.26:0/1322212826 shutdown_connections 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- 172.21.15.26:0/1322212826 wait complete. 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 Processor -- start 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.393+0000 7f846250c700 1 -- start start 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f846250c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f846250c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f845c10b9f0 con 0x7f845c108b20 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f845bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f845bfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54860/0 (socket says 172.21.15.26:54860) 2024-04-23T18:07:58.273 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f845bfff700 1 -- 172.21.15.26:0/1812652698 learned_addr learned my addr 172.21.15.26:0/1812652698 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f845bfff700 1 -- 172.21.15.26:0/1812652698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845c19f380 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f845bfff700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f844c0117b0 tx=0x7f844c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c003af0 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f844c020070 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845c19f550 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.394+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c019630 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.395+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845c19f8f0 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.395+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f844c0198a0 con 0x7f845c108b20 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.396+0000 7f84597fa700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 0x7f844803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:58.274 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.396+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f844c04db70 con 0x7f845c108b20 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.396+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845c10bb70 con 0x7f845c108b20 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.396+0000 7f845b7fe700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 0x7f844803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.396+0000 7f845b7fe700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 0x7f844803bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8444006fd0 tx=0x7f8444006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.401+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f844c023770 con 0x7f845c108b20 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.624+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f845c103cd0 con 0x7f8448039600 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.628+0000 7f84597fa700 1 -- 172.21.15.26:0/1812652698 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f845c103cd0 con 0x7f8448039600 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.630+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 msgr2=0x7f844803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.630+0000 7f846250c700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 0x7f844803bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8444006fd0 tx=0x7f8444006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.630+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 msgr2=0x7f845c07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.630+0000 7f846250c700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f844c0117b0 tx=0x7f844c011b70 comp rx=0 tx=0).stop 2024-04-23T18:07:58.275 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 shutdown_connections 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f8448039600 0x7f844803bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 --2- 172.21.15.26:0/1812652698 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f845c108b20 0x7f845c07ba60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 >> 172.21.15.26:0/1812652698 conn(0x7f845c102a60 msgr2=0x7f845c1035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 shutdown_connections 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:57.631+0000 7f846250c700 1 -- 172.21.15.26:0/1812652698 wait complete. 2024-04-23T18:07:58.276 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mgr service... 2024-04-23T18:07:58.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:58 smithi026 ceph-mon[29364]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:07:58.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:58 smithi026 ceph-mon[29364]: Saving service mon spec with placement count:5 2024-04-23T18:07:58.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:58 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:07:59.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T18:07:59.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.702+0000 7eff55585700 1 Processor -- start 2024-04-23T18:07:59.721 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.702+0000 7eff55585700 1 -- start start 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.702+0000 7eff55585700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.702+0000 7eff55585700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7eff50108f50 con 0x7eff50108560 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.702+0000 7eff4effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42776/0 (socket says 172.21.15.26:42776) 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4effd700 1 -- 172.21.15.26:0/2324789442 learned_addr learned my addr 172.21.15.26:0/2324789442 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4effd700 1 -- 172.21.15.26:0/2324789442 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff50109710 con 0x7eff50108560 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4effd700 1 --2- 172.21.15.26:0/2324789442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7eff40009e80 tx=0x7eff4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695fb0916d0532e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4dffb700 1 -- 172.21.15.26:0/2324789442 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff40017070 con 0x7eff50108560 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4dffb700 1 -- 172.21.15.26:0/2324789442 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff4000b8c0 con 0x7eff50108560 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.703+0000 7eff4dffb700 1 -- 172.21.15.26:0/2324789442 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff40004490 con 0x7eff50108560 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 -- 172.21.15.26:0/2324789442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 msgr2=0x7eff50108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 --2- 172.21.15.26:0/2324789442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7eff40009e80 tx=0x7eff4000b330 comp rx=0 tx=0).stop 2024-04-23T18:07:59.722 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 -- 172.21.15.26:0/2324789442 shutdown_connections 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 --2- 172.21.15.26:0/2324789442 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff50108980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 -- 172.21.15.26:0/2324789442 >> 172.21.15.26:0/2324789442 conn(0x7eff50103b40 msgr2=0x7eff50105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 -- 172.21.15.26:0/2324789442 shutdown_connections 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.704+0000 7eff55585700 1 -- 172.21.15.26:0/2324789442 wait complete. 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.705+0000 7eff55585700 1 Processor -- start 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.705+0000 7eff55585700 1 -- start start 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.705+0000 7eff55585700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.705+0000 7eff55585700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010cac0 con 0x7eff50108560 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.705+0000 7eff4effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff4effd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42778/0 (socket says 172.21.15.26:42778) 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff4effd700 1 -- 172.21.15.26:0/3917846796 learned_addr learned my addr 172.21.15.26:0/3917846796 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff4effd700 1 -- 172.21.15.26:0/3917846796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5007e0c0 con 0x7eff50108560 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff4effd700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7eff4000b7b0 tx=0x7eff4000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:59.723 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff40020020 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff40012430 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff40017040 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5007c2b0 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.706+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff5007c7f0 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.707+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7eff40023020 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.707+0000 7eff47fff700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 0x7eff3c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.707+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7eff4000f050 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.707+0000 7eff4e7fc700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 0x7eff3c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.708+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff500640a0 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.708+0000 7eff4e7fc700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 0x7eff3c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff38006fd0 tx=0x7eff38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.713+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff4002a020 con 0x7eff50108560 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.966+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7eff5007ceb0 con 0x7eff3c0395b0 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.971+0000 7eff47fff700 1 -- 172.21.15.26:0/3917846796 <== mgr.14120 v2:172.21.15.26:6800/2629443403 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7eff5007ceb0 con 0x7eff3c0395b0 2024-04-23T18:07:59.724 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 msgr2=0x7eff3c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 0x7eff3c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7eff38006fd0 tx=0x7eff38006e40 comp rx=0 tx=0).stop 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 msgr2=0x7eff5007db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7eff4000b7b0 tx=0x7eff4000be20 comp rx=0 tx=0).stop 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 shutdown_connections 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7eff3c0395b0 0x7eff3c03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 --2- 172.21.15.26:0/3917846796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7eff50108560 0x7eff5007db80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 >> 172.21.15.26:0/3917846796 conn(0x7eff50103b40 msgr2=0x7eff501046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 shutdown_connections 2024-04-23T18:07:59.725 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:07:58.973+0000 7eff55585700 1 -- 172.21.15.26:0/3917846796 wait complete. 2024-04-23T18:08:00.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:59 smithi026 ceph-mon[29364]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:00.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:59 smithi026 ceph-mon[29364]: Saving service mgr spec with placement count:2 2024-04-23T18:08:00.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:07:59 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:08:00.957 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980f0c9700 1 Processor -- start 2024-04-23T18:08:00.957 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980f0c9700 1 -- start start 2024-04-23T18:08:00.957 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980f0c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:00.957 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980f0c9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9808108f50 con 0x7f9808108560 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980ce65700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980ce65700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42784/0 (socket says 172.21.15.26:42784) 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.161+0000 7f980ce65700 1 -- 172.21.15.26:0/1279159870 learned_addr learned my addr 172.21.15.26:0/1279159870 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f980ce65700 1 -- 172.21.15.26:0/1279159870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9808109710 con 0x7f9808108560 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f980ce65700 1 --2- 172.21.15.26:0/1279159870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0098a0 tx=0x7f97fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e55553b8f49cf1e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f98077fe700 1 -- 172.21.15.26:0/1279159870 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97fc004160 con 0x7f9808108560 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f98077fe700 1 -- 172.21.15.26:0/1279159870 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97fc004300 con 0x7f9808108560 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f98077fe700 1 -- 172.21.15.26:0/1279159870 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97fc011af0 con 0x7f9808108560 2024-04-23T18:08:00.958 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f980f0c9700 1 -- 172.21.15.26:0/1279159870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 msgr2=0x7f9808108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.162+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1279159870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0098a0 tx=0x7f97fc009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- 172.21.15.26:0/1279159870 shutdown_connections 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1279159870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f9808108980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- 172.21.15.26:0/1279159870 >> 172.21.15.26:0/1279159870 conn(0x7f9808103b40 msgr2=0x7f9808105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- 172.21.15.26:0/1279159870 shutdown_connections 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- 172.21.15.26:0/1279159870 wait complete. 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 Processor -- start 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- start start 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.163+0000 7f980f0c9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f980810cac0 con 0x7f9808108560 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980ce65700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980ce65700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42786/0 (socket says 172.21.15.26:42786) 2024-04-23T18:08:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980ce65700 1 -- 172.21.15.26:0/1786297638 learned_addr learned my addr 172.21.15.26:0/1786297638 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980ce65700 1 -- 172.21.15.26:0/1786297638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980807e0c0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980ce65700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0117b0 tx=0x7f97fc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97fc003af0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97fc020070 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97fc019630 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980807c2b0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.164+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980807c7f0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.165+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f97fc01a9c0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.165+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98080640a0 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.165+0000 7f9805ffb700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 0x7f97f803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.166+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f97fc04db30 con 0x7f9808108560 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.166+0000 7f9807fff700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 0x7f97f803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.166+0000 7f9807fff700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 0x7f97f803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f97f4006fd0 tx=0x7f97f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:00.960 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.170+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97fc0042a0 con 0x7f9808108560 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.377+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f980807ca70 con 0x7f9808108560 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.380+0000 7f9805ffb700 1 -- 172.21.15.26:0/1786297638 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f97fc00e260 con 0x7f9808108560 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 msgr2=0x7f97f803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 0x7f97f803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f97f4006fd0 tx=0x7f97f4006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 msgr2=0x7f980807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0117b0 tx=0x7f97fc011b70 comp rx=0 tx=0).stop 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 shutdown_connections 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f97f8039600 0x7f97f803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 --2- 172.21.15.26:0/1786297638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9808108560 0x7f980807db80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 >> 172.21.15.26:0/1786297638 conn(0x7f9808103b40 msgr2=0x7f9808104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 shutdown_connections 2024-04-23T18:08:00.961 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:00.382+0000 7f980f0c9700 1 -- 172.21.15.26:0/1786297638 wait complete. 2024-04-23T18:08:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:01 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1786297638' entity='client.admin' 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.396+0000 7f39f159b700 1 Processor -- start 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.396+0000 7f39f159b700 1 -- start start 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39f159b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39f159b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec108f30 con 0x7f39ec108540 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39eaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39eaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42792/0 (socket says 172.21.15.26:42792) 2024-04-23T18:08:02.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39eaffd700 1 -- 172.21.15.26:0/532575450 learned_addr learned my addr 172.21.15.26:0/532575450 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39eaffd700 1 -- 172.21.15.26:0/532575450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec1096f0 con 0x7f39ec108540 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39eaffd700 1 --2- 172.21.15.26:0/532575450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f1c395f0fb101a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.397+0000 7f39e9ffb700 1 -- 172.21.15.26:0/532575450 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc004160 con 0x7f39ec108540 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39e9ffb700 1 -- 172.21.15.26:0/532575450 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f39dc004300 con 0x7f39ec108540 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39e9ffb700 1 -- 172.21.15.26:0/532575450 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc011af0 con 0x7f39ec108540 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 -- 172.21.15.26:0/532575450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 msgr2=0x7f39ec108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 --2- 172.21.15.26:0/532575450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 -- 172.21.15.26:0/532575450 shutdown_connections 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 --2- 172.21.15.26:0/532575450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec108960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 -- 172.21.15.26:0/532575450 >> 172.21.15.26:0/532575450 conn(0x7f39ec103b40 msgr2=0x7f39ec105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 -- 172.21.15.26:0/532575450 shutdown_connections 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.398+0000 7f39f159b700 1 -- 172.21.15.26:0/532575450 wait complete. 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39f159b700 1 Processor -- start 2024-04-23T18:08:02.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39f159b700 1 -- start start 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39f159b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39f159b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec10caa0 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39eaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39eaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42796/0 (socket says 172.21.15.26:42796) 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.399+0000 7f39eaffd700 1 -- 172.21.15.26:0/4202159237 learned_addr learned my addr 172.21.15.26:0/4202159237 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39eaffd700 1 -- 172.21.15.26:0/4202159237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec19ca70 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39eaffd700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009fb0 tx=0x7f39dc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc0036a0 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f39dc01a430 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec19cd00 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39dc019550 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.400+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec19f9f0 con 0x7f39ec108540 2024-04-23T18:08:02.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.401+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f39dc019850 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.401+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ec0640a0 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.401+0000 7f39d3fff700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 0x7f39d403fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.401+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f39dc04d900 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.401+0000 7f39ea7fc700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 0x7f39d403fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.402+0000 7f39ea7fc700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 0x7f39d403fe30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006fd0 tx=0x7f39e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.406+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39dc016dd0 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.616+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f39ec07c5d0 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.619+0000 7f39d3fff700 1 -- 172.21.15.26:0/4202159237 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) ==== 130+0+0 (secure 0 0 0) 0x7f39dc01a5e0 con 0x7f39ec108540 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 msgr2=0x7f39d403fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 0x7f39d403fe30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006fd0 tx=0x7f39e0006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:02.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 msgr2=0x7f39ec19c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009fb0 tx=0x7f39dc011940 comp rx=0 tx=0).stop 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 shutdown_connections 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f39d403d970 0x7f39d403fe30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 --2- 172.21.15.26:0/4202159237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f39ec108540 0x7f39ec19c530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 >> 172.21.15.26:0/4202159237 conn(0x7f39ec103b40 msgr2=0x7f39ec104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 shutdown_connections 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:01.621+0000 7f39f159b700 1 -- 172.21.15.26:0/4202159237 wait complete. 2024-04-23T18:08:02.350 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling the dashboard module... 2024-04-23T18:08:02.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4202159237' entity='client.admin' 2024-04-23T18:08:02.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:02 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:08:02.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:02 smithi026 ceph-mon[29364]: from='mgr.14120 172.21.15.26:0/987833025' entity='mgr.x' 2024-04-23T18:08:03.645 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1190046251' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T18:08:03.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setuser ceph since I am not root 2024-04-23T18:08:03.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setgroup ceph since I am not root 2024-04-23T18:08:03.960 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:03.811+0000 7f58020ee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:08:04.297 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:04.003+0000 7f58020ee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:08:04.297 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:04.109+0000 7f58020ee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:08:04.304 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.834+0000 7f27d0085700 1 Processor -- start 2024-04-23T18:08:04.304 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.834+0000 7f27d0085700 1 -- start start 2024-04-23T18:08:04.304 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.834+0000 7f27d0085700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.834+0000 7f27d0085700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f27c807be80 con 0x7f27c8109000 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cde21700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cde21700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42808/0 (socket says 172.21.15.26:42808) 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cde21700 1 -- 172.21.15.26:0/3218691185 learned_addr learned my addr 172.21.15.26:0/3218691185 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cde21700 1 -- 172.21.15.26:0/3218691185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c810b930 con 0x7f27c8109000 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cde21700 1 --2- 172.21.15.26:0/3218691185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098a0 tx=0x7f27c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f72384d8a0b31a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cce1f700 1 -- 172.21.15.26:0/3218691185 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c4004160 con 0x7f27c8109000 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cce1f700 1 -- 172.21.15.26:0/3218691185 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c4004300 con 0x7f27c8109000 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.835+0000 7f27cce1f700 1 -- 172.21.15.26:0/3218691185 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c4011a20 con 0x7f27c8109000 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 -- 172.21.15.26:0/3218691185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 msgr2=0x7f27c810b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 --2- 172.21.15.26:0/3218691185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098a0 tx=0x7f27c4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 -- 172.21.15.26:0/3218691185 shutdown_connections 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 --2- 172.21.15.26:0/3218691185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c810b3f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:04.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 -- 172.21.15.26:0/3218691185 >> 172.21.15.26:0/3218691185 conn(0x7f27c8102a50 msgr2=0x7f27c8104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 -- 172.21.15.26:0/3218691185 shutdown_connections 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.836+0000 7f27d0085700 1 -- 172.21.15.26:0/3218691185 wait complete. 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27d0085700 1 Processor -- start 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27d0085700 1 -- start start 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27d0085700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27d0085700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810c930 con 0x7f27c8109000 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27cde21700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27cde21700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42812/0 (socket says 172.21.15.26:42812) 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27cde21700 1 -- 172.21.15.26:0/1190046251 learned_addr learned my addr 172.21.15.26:0/1190046251 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.837+0000 7f27cde21700 1 -- 172.21.15.26:0/1190046251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c819c960 con 0x7f27c8109000 2024-04-23T18:08:04.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27cde21700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27c40116e0 tx=0x7f27c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c4003b20 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c401b8f0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c819cbf0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c40238c0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.838+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c819f8e0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.839+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f27c401baa0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.839+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c80640a0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.839+0000 7f27baffd700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 0x7f27b403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.839+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f27c404dfe0 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.839+0000 7f27cd620700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 0x7f27b403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.840+0000 7f27cd620700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 0x7f27b403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27bc006fd0 tx=0x7f27bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:02.844+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27c401f030 con 0x7f27c8109000 2024-04-23T18:08:04.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.131+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f27c819fe90 con 0x7f27c8109000 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.621+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f27c401a3d0 con 0x7f27c8109000 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.622+0000 7f27baffd700 1 -- 172.21.15.26:0/1190046251 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f27c4023a60 con 0x7f27c8109000 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 msgr2=0x7f27b403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 0x7f27b403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27bc006fd0 tx=0x7f27bc006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 msgr2=0x7f27c819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27c40116e0 tx=0x7f27c4011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 shutdown_connections 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f27b40395b0 0x7f27b403ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 --2- 172.21.15.26:0/1190046251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8109000 0x7f27c819c420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 >> 172.21.15.26:0/1190046251 conn(0x7f27c8102a50 msgr2=0x7f27c8103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 shutdown_connections 2024-04-23T18:08:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:03.624+0000 7f27d0085700 1 -- 172.21.15.26:0/1190046251 wait complete. 2024-04-23T18:08:04.547 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:04.440+0000 7f58020ee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:08:04.547 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:04.529+0000 7f58020ee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:08:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1190046251' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T18:08:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:04 smithi026 ceph-mon[29364]: mgrmap e9: x(active, since 19s) 2024-04-23T18:08:05.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3866572818' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:08:05.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff042c9e700 1 Processor -- start 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff042c9e700 1 -- start start 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff042c9e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff042c9e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c063ce0 con 0x7ff03c106e20 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff040a3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff040a3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42842/0 (socket says 172.21.15.26:42842) 2024-04-23T18:08:05.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.760+0000 7ff040a3a700 1 -- 172.21.15.26:0/583087336 learned_addr learned my addr 172.21.15.26:0/583087336 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.761+0000 7ff040a3a700 1 -- 172.21.15.26:0/583087336 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c107780 con 0x7ff03c106e20 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.761+0000 7ff040a3a700 1 --2- 172.21.15.26:0/583087336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0300098a0 tx=0x7ff030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e6b340a06c2a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.761+0000 7ff03b7fe700 1 -- 172.21.15.26:0/583087336 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff030004160 con 0x7ff03c106e20 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.761+0000 7ff03b7fe700 1 -- 172.21.15.26:0/583087336 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff030004300 con 0x7ff03c106e20 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.761+0000 7ff03b7fe700 1 -- 172.21.15.26:0/583087336 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff030011a20 con 0x7ff03c106e20 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- 172.21.15.26:0/583087336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 msgr2=0x7ff03c107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 --2- 172.21.15.26:0/583087336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff0300098a0 tx=0x7ff030009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- 172.21.15.26:0/583087336 shutdown_connections 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 --2- 172.21.15.26:0/583087336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c107240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- 172.21.15.26:0/583087336 >> 172.21.15.26:0/583087336 conn(0x7ff03c102a40 msgr2=0x7ff03c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- 172.21.15.26:0/583087336 shutdown_connections 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- 172.21.15.26:0/583087336 wait complete. 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 Processor -- start 2024-04-23T18:08:05.664 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.762+0000 7ff042c9e700 1 -- start start 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff042c9e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff042c9e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1087f0 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff040a3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff040a3a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42854/0 (socket says 172.21.15.26:42854) 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff040a3a700 1 -- 172.21.15.26:0/3866572818 learned_addr learned my addr 172.21.15.26:0/3866572818 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff040a3a700 1 -- 172.21.15.26:0/3866572818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c19eaf0 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff040a3a700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0300116e0 tx=0x7ff030011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff030003a60 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff03001b8f0 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c19ed80 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.763+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff030019860 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.764+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c1a1a70 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.764+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7ff030019b60 con 0x7ff03c106e20 2024-04-23T18:08:05.665 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.765+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03c0640a0 con 0x7ff03c106e20 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.765+0000 7ff039ffb700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 0x7ff02c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.765+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff03004dd40 con 0x7ff03c106e20 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.765+0000 7ff03bfff700 1 -- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 msgr2=0x7ff02c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.765+0000 7ff03bfff700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 0x7ff02c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.770+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff03001f030 con 0x7ff03c106e20 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.966+0000 7ff03bfff700 1 -- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 msgr2=0x7ff02c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:04.966+0000 7ff03bfff700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 0x7ff02c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.083+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ff03c02e1d0 con 0x7ff03c106e20 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.083+0000 7ff039ffb700 1 -- 172.21.15.26:0/3866572818 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7ff030029260 con 0x7ff03c106e20 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.085+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 msgr2=0x7ff02c03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.085+0000 7ff042c9e700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 0x7ff02c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.085+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 msgr2=0x7ff03c19e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:05.666 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.085+0000 7ff042c9e700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff0300116e0 tx=0x7ff030011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 shutdown_connections 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7ff02c0395b0 0x7ff02c03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 --2- 172.21.15.26:0/3866572818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff03c106e20 0x7ff03c19e5b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 >> 172.21.15.26:0/3866572818 conn(0x7ff03c102a40 msgr2=0x7ff03c195240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 shutdown_connections 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:05.086+0000 7ff042c9e700 1 -- 172.21.15.26:0/3866572818 wait complete. 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-04-23T18:08:05.667 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 9... 2024-04-23T18:08:05.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:05.790+0000 7f58020ee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:08:06.316 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:05.970+0000 7f58020ee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:08:06.316 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:06.151+0000 7f58020ee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:08:06.709 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:06.451+0000 7f58020ee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:08:06.709 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:06.547+0000 7f58020ee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:08:07.209 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:06.757+0000 7f58020ee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:08:07.210 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:06.958+0000 7f58020ee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:08:07.959 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:07.582+0000 7f58020ee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:08:08.325 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:08.021+0000 7f58020ee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:08:08.326 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:08.121+0000 7f58020ee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:08:08.326 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:08.331+0000 7f58020ee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:08:09.120 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:08.874+0000 7f58020ee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:08:09.120 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:08.970+0000 7f58020ee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:08:09.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:09.223+0000 7f58020ee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:08:09.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:09.365+0000 7f58020ee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:08:09.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:09.464+0000 7f58020ee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:08:09.709 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:09.585+0000 7f58020ee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:08:10.209 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:09.900+0000 7f58020ee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:08:10.803 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:10.530+0000 7f58020ee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:08:10.803 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:08:10.755+0000 7f58020ee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:08:11.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: Active manager daemon x restarted 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: Activating manager daemon x 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: mgrmap e10: x(active, starting, since 0.00321492s) 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: Manager daemon x is now available 2024-04-23T18:08:11.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:12.125 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:08:12.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:11 smithi026 ceph-mon[29364]: mgrmap e11: x(active, since 1.0057s) 2024-04-23T18:08:12.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:12.772 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.116+0000 7f1cd556c700 1 Processor -- start 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.116+0000 7f1cd556c700 1 -- start start 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.116+0000 7f1cd556c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.116+0000 7f1cd556c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd0108f50 con 0x7f1cd0108560 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1cceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1cceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42866/0 (socket says 172.21.15.26:42866) 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1cceffd700 1 -- 172.21.15.26:0/1184527459 learned_addr learned my addr 172.21.15.26:0/1184527459 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1cceffd700 1 -- 172.21.15.26:0/1184527459 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd0109710 con 0x7f1cd0108560 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1cceffd700 1 --2- 172.21.15.26:0/1184527459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098a0 tx=0x7f1cb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ff61134281de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1ccdffb700 1 -- 172.21.15.26:0/1184527459 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cb8004160 con 0x7f1cd0108560 2024-04-23T18:08:12.773 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1ccdffb700 1 -- 172.21.15.26:0/1184527459 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb8004300 con 0x7f1cd0108560 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.117+0000 7f1ccdffb700 1 -- 172.21.15.26:0/1184527459 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cb8011a20 con 0x7f1cd0108560 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 -- 172.21.15.26:0/1184527459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 msgr2=0x7f1cd0108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 --2- 172.21.15.26:0/1184527459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098a0 tx=0x7f1cb8009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 -- 172.21.15.26:0/1184527459 shutdown_connections 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 --2- 172.21.15.26:0/1184527459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd0108980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 -- 172.21.15.26:0/1184527459 >> 172.21.15.26:0/1184527459 conn(0x7f1cd0103b40 msgr2=0x7f1cd0105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 -- 172.21.15.26:0/1184527459 shutdown_connections 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.118+0000 7f1cd556c700 1 -- 172.21.15.26:0/1184527459 wait complete. 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cd556c700 1 Processor -- start 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cd556c700 1 -- start start 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cd556c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cd556c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd010cac0 con 0x7f1cd0108560 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:12.774 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cceffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42880/0 (socket says 172.21.15.26:42880) 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cceffd700 1 -- 172.21.15.26:0/3759182572 learned_addr learned my addr 172.21.15.26:0/3759182572 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cceffd700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd019ca00 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.119+0000 7f1cceffd700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80119f0 tx=0x7f1cb8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.120+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cb80039e0 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.120+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb801b8f0 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.120+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd019cc90 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.120+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd019f980 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.120+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cb80196d0 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f1cb8019980 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cc7fff700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc03c1e0 con 0x7f1cbc039610 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1cb804de60 con 0x7f1cd0108560 2024-04-23T18:08:12.775 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cce7fc700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.121+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.322+0000 7f1cce7fc700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.322+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.723+0000 7f1cce7fc700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:06.723+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:07.524+0000 7f1cce7fc700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:07.524+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:09.125+0000 7f1cce7fc700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/2629443403 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:09.125+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:10.759+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f1cb801a8f0 con 0x7f1cd0108560 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:10.759+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 msgr2=0x7f1cbc03bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:10.759+0000 7f1cc7fff700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.760+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f1cb804fe70 con 0x7f1cd0108560 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.761+0000 7f1cc7fff700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 0x7f1cbc03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.761+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc03c1e0 con 0x7f1cbc03cff0 2024-04-23T18:08:12.776 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.761+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 0x7f1cbc03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.762+0000 7f1cce7fc700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 0x7f1cbc03f3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0003a10 tx=0x7f1cc0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.763+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f1cbc03c1e0 con 0x7f1cbc03cff0 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.769+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1cd0069ba0 con 0x7f1cbc03cff0 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.769+0000 7f1cc7fff700 1 -- 172.21.15.26:0/3759182572 <== mgr.14152 v2:172.21.15.26:6800/270207440 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f1cd0069ba0 con 0x7f1cbc03cff0 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 msgr2=0x7f1cbc03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 0x7f1cbc03f3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0003a10 tx=0x7f1cc0009270 comp rx=0 tx=0).stop 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 msgr2=0x7f1cd019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80119f0 tx=0x7f1cb8011ad0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 shutdown_connections 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1cbc03cff0 0x7f1cbc03f3e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1cd0108560 0x7f1cd019c4c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 --2- 172.21.15.26:0/3759182572 >> [v2:172.21.15.26:6800/2629443403,v1:172.21.15.26:6801/2629443403] conn(0x7f1cbc039610 0x7f1cbc03bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:12.777 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 >> 172.21.15.26:0/3759182572 conn(0x7f1cd0103b40 msgr2=0x7f1cd01046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:12.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 shutdown_connections 2024-04-23T18:08:12.778 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:11.770+0000 7f1cd556c700 1 -- 172.21.15.26:0/3759182572 wait complete. 2024-04-23T18:08:12.778 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 9 is available 2024-04-23T18:08:12.778 INFO:teuthology.orchestra.run.smithi026.stdout:Generating a dashboard self-signed certificate... 2024-04-23T18:08:13.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:08:11] ENGINE Bus STARTING 2024-04-23T18:08:13.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:08:11] ENGINE Serving on https://172.21.15.26:7150 2024-04-23T18:08:13.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:08:12] ENGINE Serving on http://172.21.15.26:8765 2024-04-23T18:08:13.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:08:12] ENGINE Bus STARTED 2024-04-23T18:08:14.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:13 smithi026 ceph-mon[29364]: mgrmap e12: x(active, since 2s) 2024-04-23T18:08:14.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:14.085 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:13 smithi026 ceph-mon[29364]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:14.085 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:14.085 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:14.087 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.258+0000 7f154ad18700 1 Processor -- start 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.258+0000 7f154ad18700 1 -- start start 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.258+0000 7f154ad18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.258+0000 7f154ad18700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1544063d00 con 0x7f1544109010 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f1548ab4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:14.092 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f1548ab4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43724/0 (socket says 172.21.15.26:43724) 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f1548ab4700 1 -- 172.21.15.26:0/638385549 learned_addr learned my addr 172.21.15.26:0/638385549 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f1548ab4700 1 -- 172.21.15.26:0/638385549 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154410b940 con 0x7f1544109010 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f1548ab4700 1 --2- 172.21.15.26:0/638385549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15380098a0 tx=0x7f1538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e624e2a066a52074 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f15437fe700 1 -- 172.21.15.26:0/638385549 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1538004160 con 0x7f1544109010 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f15437fe700 1 -- 172.21.15.26:0/638385549 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1538004300 con 0x7f1544109010 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.259+0000 7f15437fe700 1 -- 172.21.15.26:0/638385549 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1538011a20 con 0x7f1544109010 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 -- 172.21.15.26:0/638385549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 msgr2=0x7f154410b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 --2- 172.21.15.26:0/638385549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15380098a0 tx=0x7f1538009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 -- 172.21.15.26:0/638385549 shutdown_connections 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 --2- 172.21.15.26:0/638385549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154410b400 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 -- 172.21.15.26:0/638385549 >> 172.21.15.26:0/638385549 conn(0x7f1544102a40 msgr2=0x7f1544104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 -- 172.21.15.26:0/638385549 shutdown_connections 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.260+0000 7f154ad18700 1 -- 172.21.15.26:0/638385549 wait complete. 2024-04-23T18:08:14.093 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f154ad18700 1 Processor -- start 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f154ad18700 1 -- start start 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f154ad18700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f154ad18700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f154410c8f0 con 0x7f1544109010 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1548ab4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1548ab4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43728/0 (socket says 172.21.15.26:43728) 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1548ab4700 1 -- 172.21.15.26:0/1891385391 learned_addr learned my addr 172.21.15.26:0/1891385391 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1548ab4700 1 -- 172.21.15.26:0/1891385391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154419c930 con 0x7f1544109010 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1548ab4700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15380116e0 tx=0x7f1538011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.261+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1538003a60 con 0x7f1544109010 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.262+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153801b8f0 con 0x7f1544109010 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.262+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154419cbc0 con 0x7f1544109010 2024-04-23T18:08:14.094 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.262+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1538019860 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.262+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154419f8b0 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.263+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15440640a0 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.263+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f1538019b10 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.263+0000 7f1541ffb700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 0x7f153403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.263+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f153804ddc0 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.263+0000 7f1543fff700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 0x7f153403bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.264+0000 7f1543fff700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 0x7f153403bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1530006fd0 tx=0x7f1530006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.268+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f153801f030 con 0x7f1544109010 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.486+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f1544103ba0 con 0x7f1534039610 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.532+0000 7f1541ffb700 1 -- 172.21.15.26:0/1891385391 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f1544103ba0 con 0x7f1534039610 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 msgr2=0x7f153403bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 0x7f153403bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1530006fd0 tx=0x7f1530006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 msgr2=0x7f154419c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15380116e0 tx=0x7f1538011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:14.095 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 shutdown_connections 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1534039610 0x7f153403bad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 --2- 172.21.15.26:0/1891385391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1544109010 0x7f154419c3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 >> 172.21.15.26:0/1891385391 conn(0x7f1544102a40 msgr2=0x7f1544103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 shutdown_connections 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:13.534+0000 7f154ad18700 1 -- 172.21.15.26:0/1891385391 wait complete. 2024-04-23T18:08:14.096 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial admin user... 2024-04-23T18:08:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:15.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0qUkWp1kmo3wG/9P2Fme/ORJytW3lQY/CQi6S/10EFyd6o9NoEzy.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713895695, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba80d0700 1 Processor -- start 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba80d0700 1 -- start start 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba80d0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba80d0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0063d60 con 0x7f0ba0108fb0 2024-04-23T18:08:16.305 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba5e6c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba5e6c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43742/0 (socket says 172.21.15.26:43742) 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.552+0000 7f0ba5e6c700 1 -- 172.21.15.26:0/3500179181 learned_addr learned my addr 172.21.15.26:0/3500179181 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.553+0000 7f0ba5e6c700 1 -- 172.21.15.26:0/3500179181 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba010b8e0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.553+0000 7f0ba5e6c700 1 --2- 172.21.15.26:0/3500179181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b900098a0 tx=0x7f0b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34da8eb89e366458 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.553+0000 7f0ba4e6a700 1 -- 172.21.15.26:0/3500179181 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b90004160 con 0x7f0ba0108fb0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.553+0000 7f0ba4e6a700 1 -- 172.21.15.26:0/3500179181 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b90004300 con 0x7f0ba0108fb0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.553+0000 7f0ba4e6a700 1 -- 172.21.15.26:0/3500179181 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b90011a20 con 0x7f0ba0108fb0 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3500179181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 msgr2=0x7f0ba010b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3500179181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b900098a0 tx=0x7f0b90009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3500179181 shutdown_connections 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3500179181 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba010b3a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3500179181 >> 172.21.15.26:0/3500179181 conn(0x7f0ba0102a40 msgr2=0x7f0ba0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:16.306 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3500179181 shutdown_connections 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3500179181 wait complete. 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 Processor -- start 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.554+0000 7f0ba80d0700 1 -- start start 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba80d0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba80d0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba010c8f0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba5e6c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba5e6c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43752/0 (socket says 172.21.15.26:43752) 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba5e6c700 1 -- 172.21.15.26:0/3688196794 learned_addr learned my addr 172.21.15.26:0/3688196794 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba5e6c700 1 -- 172.21.15.26:0/3688196794 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba019c930 con 0x7f0ba0108fb0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba5e6c700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b900116e0 tx=0x7f0b90011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b90003a60 con 0x7f0ba0108fb0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9001b8f0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba019cbc0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.307 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba019f8b0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.555+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b90019860 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.556+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f0b90019b50 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.557+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b84005360 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.557+0000 7f0b96ffd700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 0x7f0b8c03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.557+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0b9004dda0 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.557+0000 7f0ba566b700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 0x7f0b8c03bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.557+0000 7f0ba566b700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 0x7f0b8c03bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c006fd0 tx=0x7f0b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.562+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b9001f030 con 0x7f0ba0108fb0 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:14.810+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f0b84002430 con 0x7f0b8c039610 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.057+0000 7f0b96ffd700 1 -- 172.21.15.26:0/3688196794 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f0b84002430 con 0x7f0b8c039610 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 msgr2=0x7f0b8c03bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 0x7f0b8c03bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c006fd0 tx=0x7f0b9c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 msgr2=0x7f0ba019c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b900116e0 tx=0x7f0b90011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 shutdown_connections 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0b8c039610 0x7f0b8c03bad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 --2- 172.21.15.26:0/3688196794 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ba0108fb0 0x7f0ba019c3f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.059+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 >> 172.21.15.26:0/3688196794 conn(0x7f0ba0102a40 msgr2=0x7f0ba0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.060+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 shutdown_connections 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:15.060+0000 7f0ba80d0700 1 -- 172.21.15.26:0/3688196794 wait complete. 2024-04-23T18:08:16.309 INFO:teuthology.orchestra.run.smithi026.stdout:Fetching dashboard port number... 2024-04-23T18:08:16.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:16 smithi026 ceph-mon[29364]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2269905624' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T18:08:17.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T18:08:17.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.760+0000 7f2cff8be700 1 Processor -- start 2024-04-23T18:08:17.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.760+0000 7f2cff8be700 1 -- start start 2024-04-23T18:08:17.553 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.760+0000 7f2cff8be700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.760+0000 7f2cff8be700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf8108f50 con 0x7f2cf8108560 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2cfd65a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2cfd65a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43762/0 (socket says 172.21.15.26:43762) 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2cfd65a700 1 -- 172.21.15.26:0/1171388073 learned_addr learned my addr 172.21.15.26:0/1171388073 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2cfd65a700 1 -- 172.21.15.26:0/1171388073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf8109710 con 0x7f2cf8108560 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2cfd65a700 1 --2- 172.21.15.26:0/1171388073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40098a0 tx=0x7f2cf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce55e3eb98cb9e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2ceffff700 1 -- 172.21.15.26:0/1171388073 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf4004160 con 0x7f2cf8108560 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2ceffff700 1 -- 172.21.15.26:0/1171388073 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cf4004300 con 0x7f2cf8108560 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.761+0000 7f2ceffff700 1 -- 172.21.15.26:0/1171388073 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf4011a20 con 0x7f2cf8108560 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 -- 172.21.15.26:0/1171388073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 msgr2=0x7f2cf8108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 --2- 172.21.15.26:0/1171388073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40098a0 tx=0x7f2cf4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 -- 172.21.15.26:0/1171388073 shutdown_connections 2024-04-23T18:08:17.554 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 --2- 172.21.15.26:0/1171388073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf8108980 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 -- 172.21.15.26:0/1171388073 >> 172.21.15.26:0/1171388073 conn(0x7f2cf8103b40 msgr2=0x7f2cf8105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 -- 172.21.15.26:0/1171388073 shutdown_connections 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.762+0000 7f2cff8be700 1 -- 172.21.15.26:0/1171388073 wait complete. 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cff8be700 1 Processor -- start 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cff8be700 1 -- start start 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cff8be700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cff8be700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810cac0 con 0x7f2cf8108560 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cfd65a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cfd65a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43770/0 (socket says 172.21.15.26:43770) 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cfd65a700 1 -- 172.21.15.26:0/2269905624 learned_addr learned my addr 172.21.15.26:0/2269905624 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cfd65a700 1 -- 172.21.15.26:0/2269905624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf807e0c0 con 0x7f2cf8108560 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.763+0000 7f2cfd65a700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c00 tx=0x7f2cf4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf4003aa0 con 0x7f2cf8108560 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cf401b8f0 con 0x7f2cf8108560 2024-04-23T18:08:17.555 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf4023890 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf807c2b0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf807c7f0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.764+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f2cf402c3f0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.765+0000 7f2cee7fc700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 0x7f2ce403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.765+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2cf404dfb0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.765+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf80640a0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.765+0000 7f2cfce59700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 0x7f2ce403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.765+0000 7f2cfce59700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 0x7f2ce403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006fd0 tx=0x7f2ce8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.770+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cf401f030 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.977+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f2cf807ca70 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.977+0000 7f2cee7fc700 1 -- 172.21.15.26:0/2269905624 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f2cf40202b0 con 0x7f2cf8108560 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 msgr2=0x7f2ce403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:17.556 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 0x7f2ce403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006fd0 tx=0x7f2ce8006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 msgr2=0x7f2cf807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c00 tx=0x7f2cf4011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 shutdown_connections 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2ce4039600 0x7f2ce403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 --2- 172.21.15.26:0/2269905624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cf8108560 0x7f2cf807db80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.979+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 >> 172.21.15.26:0/2269905624 conn(0x7f2cf8103b40 msgr2=0x7f2cf8104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.980+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 shutdown_connections 2024-04-23T18:08:17.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:16.980+0000 7f2cff8be700 1 -- 172.21.15.26:0/2269905624 wait complete. 2024-04-23T18:08:17.562 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:08:17.562 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-04-23T18:08:17.568 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:08:17.568 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-04-23T18:08:17.568 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-04-23T18:08:17.568 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph Dashboard is now available at: 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout: URL: https://smithi026:8443/ 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout: User: admin 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout: Password: 3ka1h290t5 2024-04-23T18:08:17.570 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:17.571 INFO:teuthology.orchestra.run.smithi026.stdout:Saving cluster configuration to /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config directory 2024-04-23T18:08:17.571 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling autotune for osd_memory_target 2024-04-23T18:08:18.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:18 smithi026 ceph-mon[29364]: mgrmap e13: x(active, since 6s) 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.967+0000 7f353e099700 1 Processor -- start 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.967+0000 7f353e099700 1 -- start start 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.967+0000 7f353e099700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.967+0000 7f353e099700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3538108f50 con 0x7f3538108560 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f35377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f35377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43778/0 (socket says 172.21.15.26:43778) 2024-04-23T18:08:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f35377fe700 1 -- 172.21.15.26:0/4107592923 learned_addr learned my addr 172.21.15.26:0/4107592923 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f35377fe700 1 -- 172.21.15.26:0/4107592923 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3538109710 con 0x7f3538108560 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f35377fe700 1 --2- 172.21.15.26:0/4107592923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35280098a0 tx=0x7f3528009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bcaf3658359941 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f3536ffd700 1 -- 172.21.15.26:0/4107592923 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3528004160 con 0x7f3538108560 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f3536ffd700 1 -- 172.21.15.26:0/4107592923 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3528004300 con 0x7f3538108560 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.968+0000 7f3536ffd700 1 -- 172.21.15.26:0/4107592923 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3528011a20 con 0x7f3538108560 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 -- 172.21.15.26:0/4107592923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 msgr2=0x7f3538108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 --2- 172.21.15.26:0/4107592923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35280098a0 tx=0x7f3528009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 -- 172.21.15.26:0/4107592923 shutdown_connections 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 --2- 172.21.15.26:0/4107592923 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f3538108980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 -- 172.21.15.26:0/4107592923 >> 172.21.15.26:0/4107592923 conn(0x7f3538103b40 msgr2=0x7f3538105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 -- 172.21.15.26:0/4107592923 shutdown_connections 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.969+0000 7f353e099700 1 -- 172.21.15.26:0/4107592923 wait complete. 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f353e099700 1 Processor -- start 2024-04-23T18:08:18.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f353e099700 1 -- start start 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f353e099700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f353e099700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f353810cac0 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35377fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58626/0 (socket says 172.21.15.26:58626) 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35377fe700 1 -- 172.21.15.26:0/19281319 learned_addr learned my addr 172.21.15.26:0/19281319 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35377fe700 1 -- 172.21.15.26:0/19281319 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353819c9c0 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35377fe700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3528000c00 tx=0x7f3528011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3528003aa0 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f352801b8f0 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.970+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353819cc50 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.971+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3528023890 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.971+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353819f940 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.971+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f352801b440 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.971+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35380640a0 con 0x7f3538108560 2024-04-23T18:08:18.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.972+0000 7f35357fa700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 0x7f352403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.972+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f352804e090 con 0x7f3538108560 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.972+0000 7f352ffff700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 0x7f352403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.972+0000 7f352ffff700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 0x7f352403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3520006fd0 tx=0x7f3520006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:17.977+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f352801f030 con 0x7f3538108560 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.186+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f353819fb10 con 0x7f3538108560 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.186+0000 7f35357fa700 1 -- 172.21.15.26:0/19281319 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f353819fb10 con 0x7f3538108560 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 msgr2=0x7f352403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 0x7f352403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3520006fd0 tx=0x7f3520006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 msgr2=0x7f353819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3528000c00 tx=0x7f3528011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 shutdown_connections 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3524039700 0x7f352403bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.762 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 --2- 172.21.15.26:0/19281319 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3538108560 0x7f353819c480 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:18.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 >> 172.21.15.26:0/19281319 conn(0x7f3538103b40 msgr2=0x7f35381046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:18.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 shutdown_connections 2024-04-23T18:08:18.763 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:18.188+0000 7f353e099700 1 -- 172.21.15.26:0/19281319 wait complete. 2024-04-23T18:08:20.024 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.141+0000 7ff46da96700 1 Processor -- start 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.141+0000 7ff46da96700 1 -- start start 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff46da96700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff46da96700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff46807c900 con 0x7ff46807de80 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff4677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff4677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58636/0 (socket says 172.21.15.26:58636) 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff4677fe700 1 -- 172.21.15.26:0/4038435518 learned_addr learned my addr 172.21.15.26:0/4038435518 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff4677fe700 1 -- 172.21.15.26:0/4038435518 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46807ca80 con 0x7ff46807de80 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.142+0000 7ff4677fe700 1 --2- 172.21.15.26:0/4038435518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff4580098a0 tx=0x7ff458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c012f43d3aac4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff4667fc700 1 -- 172.21.15.26:0/4038435518 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff458004160 con 0x7ff46807de80 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff4667fc700 1 -- 172.21.15.26:0/4038435518 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff458004300 con 0x7ff46807de80 2024-04-23T18:08:20.025 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff4667fc700 1 -- 172.21.15.26:0/4038435518 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff458011a20 con 0x7ff46807de80 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 -- 172.21.15.26:0/4038435518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 msgr2=0x7ff46807c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 --2- 172.21.15.26:0/4038435518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff4580098a0 tx=0x7ff458009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 -- 172.21.15.26:0/4038435518 shutdown_connections 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 --2- 172.21.15.26:0/4038435518 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46807c330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 -- 172.21.15.26:0/4038435518 >> 172.21.15.26:0/4038435518 conn(0x7ff468103b40 msgr2=0x7ff468105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 -- 172.21.15.26:0/4038435518 shutdown_connections 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.143+0000 7ff46da96700 1 -- 172.21.15.26:0/4038435518 wait complete. 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff46da96700 1 Processor -- start 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff46da96700 1 -- start start 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff46da96700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff46da96700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810eae0 con 0x7ff46807de80 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff4677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff4677fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58646/0 (socket says 172.21.15.26:58646) 2024-04-23T18:08:20.026 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.144+0000 7ff4677fe700 1 -- 172.21.15.26:0/3168483161 learned_addr learned my addr 172.21.15.26:0/3168483161 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff4677fe700 1 -- 172.21.15.26:0/3168483161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46819eb90 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff4677fe700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4580116e0 tx=0x7ff458011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff458003b20 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff45801b8f0 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46819ee20 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4580238c0 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.145+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4681a1b10 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.146+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff458023a60 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.146+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4680640a0 con 0x7ff46807de80 2024-04-23T18:08:20.027 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.146+0000 7ff464ff9700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 0x7ff45003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.146+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff45804e210 con 0x7ff46807de80 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.146+0000 7ff466ffd700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 0x7ff45003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.147+0000 7ff466ffd700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 0x7ff45003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff45c006fd0 tx=0x7ff45c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.151+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff45801f030 con 0x7ff46807de80 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.474+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7ff4681a20c0 con 0x7ff46807de80 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.476+0000 7ff464ff9700 1 -- 172.21.15.26:0/3168483161 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7ff45802bb90 con 0x7ff46807de80 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.478+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 msgr2=0x7ff45003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.478+0000 7ff46da96700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 0x7ff45003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff45c006fd0 tx=0x7ff45c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.478+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 msgr2=0x7ff46819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.478+0000 7ff46da96700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4580116e0 tx=0x7ff458011aa0 comp rx=0 tx=0).stop 2024-04-23T18:08:20.028 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 shutdown_connections 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff4500396b0 0x7ff45003bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 --2- 172.21.15.26:0/3168483161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff46807de80 0x7ff46819e650 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 >> 172.21.15.26:0/3168483161 conn(0x7ff468103b40 msgr2=0x7ff4681046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 shutdown_connections 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:19.479+0000 7ff46da96700 1 -- 172.21.15.26:0/3168483161 wait complete. 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.029 INFO:teuthology.orchestra.run.smithi026.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: ceph telemetry on 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout:For more information see: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:20.030 INFO:teuthology.orchestra.run.smithi026.stdout:Bootstrap complete. 2024-04-23T18:08:20.066 INFO:tasks.cephadm:Fetching config... 2024-04-23T18:08:20.066 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:08:20.067 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T18:08:20.083 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T18:08:20.083 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:08:20.083 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T18:08:20.138 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T18:08:20.139 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:08:20.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T18:08:20.204 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T18:08:20.205 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:08:20.205 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T18:08:20.261 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T18:08:20.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:08:20.357 INFO:teuthology.orchestra.run.smithi026.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:20.370 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:08:20.409 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:20.422 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:08:20.462 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDatsLKbrVLmrzYTMCrlz3L565NW93ZGfSr9fMxasC2yhqllPMF2jb8etsrIapR7ro5CJDdnelOsTRk+pBQmsm3Al62UnELa6Ygc98VaUdVIgnbuyglG8Ks2JD0o8mdzW/qnEeH/ojugZGoCqDnlPIpOZ6ClSa2plNAOeRo9fB1XX1WWMV53FH2Go4CWuuja4tmFq1JFVQ9eOY4D8X85Z+HZ2DHxxljEt7+Zj3yS+Ub8J0PHJzFTOY1EDfc6VCM2CaCgrn+i7ioquKiUJWCX6NzBziQ9hFDc4SCJh6gHoNQIkKnBcBX7PoP2l04VBHjw1+vB2adbo6rOBBFEZXivsfUGzeOulnQP7ume4MOi4UDRyRbi1z6qnDmHETEZi4kNSuGW0uydJpEJ8m9uU7Cc+M2H2U9vS7NAj+QtbaAh4CawgY4P7VdiMtQk7IZ1bjF+/zjvsat7Yh5f5udR2sGXVeRcVaUDkxqWvNr4Mca2oF2p+F0qbYJtQiLOFB65OKunt8= ceph-2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:20.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T18:08:20.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:20 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3168483161' entity='client.admin' 2024-04-23T18:08:20.727 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:22.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- 172.21.15.26:0/2397743025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 msgr2=0x7f0dcc105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 --2- 172.21.15.26:0/2397743025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc105340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009910 tx=0x7f0dbc009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- 172.21.15.26:0/2397743025 shutdown_connections 2024-04-23T18:08:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 --2- 172.21.15.26:0/2397743025 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc105340 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- 172.21.15.26:0/2397743025 >> 172.21.15.26:0/2397743025 conn(0x7f0dcc0fc980 msgr2=0x7f0dcc0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:22.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- 172.21.15.26:0/2397743025 shutdown_connections 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- 172.21.15.26:0/2397743025 wait complete. 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 Processor -- start 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- start start 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.846+0000 7f0dd2437700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc1083a0 con 0x7f0dcc102f50 2024-04-23T18:08:22.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58674/0 (socket says 172.21.15.26:58674) 2024-04-23T18:08:22.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dcbfff700 1 -- 172.21.15.26:0/1836626992 learned_addr learned my addr 172.21.15.26:0/1836626992 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dcbfff700 1 -- 172.21.15.26:0/1836626992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc102d10 con 0x7f0dcc102f50 2024-04-23T18:08:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dcbfff700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005b40 tx=0x7f0dbc004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dbc027070 con 0x7f0dcc102f50 2024-04-23T18:08:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc100f00 con 0x7f0dcc102f50 2024-04-23T18:08:22.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dbc00be10 con 0x7f0dcc102f50 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dbc022de0 con 0x7f0dcc102f50 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.847+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc101440 con 0x7f0dcc102f50 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.848+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0dbc02a070 con 0x7f0dcc102f50 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.848+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc108b20 con 0x7f0dcc102f50 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.848+0000 7f0dc97fa700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 0x7f0db403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:22.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.848+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0dbc01e030 con 0x7f0dcc102f50 2024-04-23T18:08:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.849+0000 7f0dcb7fe700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 0x7f0db403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:22.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.849+0000 7f0dcb7fe700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 0x7f0db403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0006fd0 tx=0x7f0dc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:22.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:22.854+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dbc013070 con 0x7f0dcc102f50 2024-04-23T18:08:23.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.060+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f0dcc191060 con 0x7f0dcc102f50 2024-04-23T18:08:23.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.062+0000 7f0dc97fa700 1 -- 172.21.15.26:0/1836626992 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f0dbc0292c0 con 0x7f0dcc102f50 2024-04-23T18:08:23.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.064+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 msgr2=0x7f0db403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.064+0000 7f0dd2437700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 0x7f0db403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0006fd0 tx=0x7f0dc0006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.064+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 msgr2=0x7f0dcc1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.064+0000 7f0dd2437700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005b40 tx=0x7f0dbc004f80 comp rx=0 tx=0).stop 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 shutdown_connections 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0db4039650 0x7f0db403bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 --2- 172.21.15.26:0/1836626992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0dcc102f50 0x7f0dcc1027d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 >> 172.21.15.26:0/1836626992 conn(0x7f0dcc0fc980 msgr2=0x7f0dcc0fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 shutdown_connections 2024-04-23T18:08:23.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:23.065+0000 7f0dd2437700 1 -- 172.21.15.26:0/1836626992 wait complete. 2024-04-23T18:08:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:23.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:23.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:23.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:23.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:23.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:23.681 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T18:08:23.681 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T18:08:23.924 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:24.323 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:24 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1836626992' entity='client.admin' 2024-04-23T18:08:24.323 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:24.323 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:25.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 -- 172.21.15.26:0/3239471983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 msgr2=0x7f00481028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:25.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 --2- 172.21.15.26:0/3239471983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f00481028c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0038009910 tx=0x7f0038009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:25.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 -- 172.21.15.26:0/3239471983 shutdown_connections 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 --2- 172.21.15.26:0/3239471983 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f00481028c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 -- 172.21.15.26:0/3239471983 >> 172.21.15.26:0/3239471983 conn(0x7f00480fda40 msgr2=0x7f00480ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 -- 172.21.15.26:0/3239471983 shutdown_connections 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.352+0000 7f004d32e700 1 -- 172.21.15.26:0/3239471983 wait complete. 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f004d32e700 1 Processor -- start 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f004d32e700 1 -- start start 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f004d32e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f004d32e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f00481083e0 con 0x7f00481024a0 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f0046ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f0046ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58690/0 (socket says 172.21.15.26:58690) 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f0046ffd700 1 -- 172.21.15.26:0/1933690875 learned_addr learned my addr 172.21.15.26:0/1933690875 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.353+0000 7f0046ffd700 1 -- 172.21.15.26:0/1933690875 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0048198030 con 0x7f00481024a0 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f0046ffd700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0038005800 tx=0x7f0038009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0038027070 con 0x7f00481024a0 2024-04-23T18:08:25.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f003800be10 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f003800f460 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481982c0 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.354+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004819afb0 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f003802a070 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0048108b60 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f002ffff700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 0x7f003003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f003801e030 con 0x7f00481024a0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f00467fc700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 0x7f003003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:25.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.355+0000 7f00467fc700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 0x7f003003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:25.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.360+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0038013070 con 0x7f00481024a0 2024-04-23T18:08:25.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.570+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f004802e090 con 0x7f00300396b0 2024-04-23T18:08:25.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.573+0000 7f002ffff700 1 -- 172.21.15.26:0/1933690875 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f004802e090 con 0x7f00300396b0 2024-04-23T18:08:25.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.575+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 msgr2=0x7f003003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:25.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.575+0000 7f004d32e700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 0x7f003003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:25.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.575+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 msgr2=0x7f0048197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:25.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.575+0000 7f004d32e700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0038005800 tx=0x7f0038009f60 comp rx=0 tx=0).stop 2024-04-23T18:08:25.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 shutdown_connections 2024-04-23T18:08:25.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f00300396b0 0x7f003003bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:25.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 --2- 172.21.15.26:0/1933690875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f00481024a0 0x7f0048197af0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:25.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 >> 172.21.15.26:0/1933690875 conn(0x7f00480fda40 msgr2=0x7f00480fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:25.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 shutdown_connections 2024-04-23T18:08:25.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:25.576+0000 7f004d32e700 1 -- 172.21.15.26:0/1933690875 wait complete. 2024-04-23T18:08:26.310 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2024-04-23T18:08:26.310 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:08:26.310 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:08:26.329 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:08:26.329 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:26.388 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2024-04-23T18:08:26.388 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch host add smithi110 2024-04-23T18:08:26.634 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:26.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:27.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:27 smithi026 ceph-mon[29364]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:08:27.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:27 smithi026 ceph-mon[29364]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:27.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:27 smithi026 ceph-mon[29364]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.247+0000 7fee7d34b700 1 -- 172.21.15.26:0/2334850127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 msgr2=0x7fee781088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.247+0000 7fee7d34b700 1 --2- 172.21.15.26:0/2334850127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee781088c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fee68009910 tx=0x7fee68009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.247+0000 7fee7d34b700 1 -- 172.21.15.26:0/2334850127 shutdown_connections 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.247+0000 7fee7d34b700 1 --2- 172.21.15.26:0/2334850127 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee781088c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.247+0000 7fee7d34b700 1 -- 172.21.15.26:0/2334850127 >> 172.21.15.26:0/2334850127 conn(0x7fee780fdf80 msgr2=0x7fee781003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.248+0000 7fee7d34b700 1 -- 172.21.15.26:0/2334850127 shutdown_connections 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.248+0000 7fee7d34b700 1 -- 172.21.15.26:0/2334850127 wait complete. 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.248+0000 7fee7d34b700 1 Processor -- start 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.248+0000 7fee7d34b700 1 -- start start 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee7d34b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:28.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee7d34b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810c940 con 0x7fee781084e0 2024-04-23T18:08:28.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee76ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:28.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee76ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48168/0 (socket says 172.21.15.26:48168) 2024-04-23T18:08:28.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee76ffd700 1 -- 172.21.15.26:0/1955229627 learned_addr learned my addr 172.21.15.26:0/1955229627 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:28.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee76ffd700 1 -- 172.21.15.26:0/1955229627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7819a600 con 0x7fee781084e0 2024-04-23T18:08:28.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee76ffd700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee680056d0 tx=0x7fee680057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:28.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee68026070 con 0x7fee781084e0 2024-04-23T18:08:28.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6800bba0 con 0x7fee781084e0 2024-04-23T18:08:28.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7819a8f0 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee68003920 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.249+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7819e990 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.250+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fee68029070 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.250+0000 7fee5ffff700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 0x7fee6003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.250+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee78103b90 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.250+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fee6801e070 con 0x7fee781084e0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.251+0000 7fee767fc700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 0x7fee6003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.251+0000 7fee767fc700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 0x7fee6003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee6c006fd0 tx=0x7fee6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:28.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.256+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee68030070 con 0x7fee781084e0 2024-04-23T18:08:28.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:28.466+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}) -- 0x7fee7819b420 con 0x7fee60039650 2024-04-23T18:08:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:28 smithi026 ceph-mon[29364]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:30.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:29 smithi026 ceph-mon[29364]: Deploying cephadm binary to smithi110 2024-04-23T18:08:32.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:31 smithi026 ceph-mon[29364]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:32.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.740+0000 7fee5ffff700 1 -- 172.21.15.26:0/1955229627 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fee7819b420 con 0x7fee60039650 2024-04-23T18:08:32.736 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi110' with addr '172.21.15.110' 2024-04-23T18:08:32.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 msgr2=0x7fee6003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 0x7fee6003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee6c006fd0 tx=0x7fee6c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 msgr2=0x7fee7819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee680056d0 tx=0x7fee680057b0 comp rx=0 tx=0).stop 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 shutdown_connections 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fee60039650 0x7fee6003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 --2- 172.21.15.26:0/1955229627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee781084e0 0x7fee7819a0c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 >> 172.21.15.26:0/1955229627 conn(0x7fee780fdf80 msgr2=0x7fee78109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:32.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 shutdown_connections 2024-04-23T18:08:32.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:32.744+0000 7fee7d34b700 1 -- 172.21.15.26:0/1955229627 wait complete. 2024-04-23T18:08:33.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:08:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:34.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:34.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:33 smithi026 ceph-mon[29364]: Added host smithi110 2024-04-23T18:08:34.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:34.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:33 smithi026 ceph-mon[29364]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:34.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:35.041 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.261+0000 7f3b2041a700 1 -- 172.21.15.26:0/2067453866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 msgr2=0x7f3b18105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.261+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2067453866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18105e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009910 tx=0x7f3b14009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.261+0000 7f3b2041a700 1 -- 172.21.15.26:0/2067453866 shutdown_connections 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.261+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2067453866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18105e70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.261+0000 7f3b2041a700 1 -- 172.21.15.26:0/2067453866 >> 172.21.15.26:0/2067453866 conn(0x7f3b18078970 msgr2=0x7f3b18078d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 -- 172.21.15.26:0/2067453866 shutdown_connections 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 -- 172.21.15.26:0/2067453866 wait complete. 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 Processor -- start 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 -- start start 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b2041a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1810b870 con 0x7f3b18105a90 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b1e1b6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b1e1b6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48190/0 (socket says 172.21.15.26:48190) 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.262+0000 7f3b1e1b6700 1 -- 172.21.15.26:0/2117942503 learned_addr learned my addr 172.21.15.26:0/2117942503 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b1e1b6700 1 -- 172.21.15.26:0/2117942503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b181973b0 con 0x7f3b18105a90 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b1e1b6700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b140056d0 tx=0x7f3b140057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b14028070 con 0x7f3b18105a90 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b18197640 con 0x7f3b18105a90 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14004370 con 0x7f3b18105a90 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b140039b0 con 0x7f3b18105a90 2024-04-23T18:08:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.263+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1819b740 con 0x7f3b18105a90 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.264+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3b1402b070 con 0x7f3b18105a90 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.264+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b181021e0 con 0x7f3b18105a90 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.264+0000 7f3b0f7fe700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 0x7f3b0403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.264+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3b14020070 con 0x7f3b18105a90 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.264+0000 7f3b1d9b5700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 0x7f3b0403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.265+0000 7f3b1d9b5700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 0x7f3b0403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b08006fd0 tx=0x7f3b08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.269+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b14013070 con 0x7f3b18105a90 2024-04-23T18:08:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.482+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b180640a0 con 0x7f3b040396b0 2024-04-23T18:08:35.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.483+0000 7f3b0f7fe700 1 -- 172.21.15.26:0/2117942503 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f3b180640a0 con 0x7f3b040396b0 2024-04-23T18:08:35.477 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:35.477 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 msgr2=0x7f3b0403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 0x7f3b0403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b08006fd0 tx=0x7f3b08006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 msgr2=0x7f3b18196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b140056d0 tx=0x7f3b140057b0 comp rx=0 tx=0).stop 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 shutdown_connections 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3b040396b0 0x7f3b0403bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 --2- 172.21.15.26:0/2117942503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b18105a90 0x7f3b18196e70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 >> 172.21.15.26:0/2117942503 conn(0x7f3b18078970 msgr2=0x7f3b180fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 shutdown_connections 2024-04-23T18:08:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:35.485+0000 7f3b2041a700 1 -- 172.21.15.26:0/2117942503 wait complete. 2024-04-23T18:08:36.092 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi131 2024-04-23T18:08:36.092 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:08:36.093 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:08:36.111 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:08:36.111 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:36.168 INFO:tasks.cephadm:Adding host smithi131 to orchestrator... 2024-04-23T18:08:36.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch host add smithi131 2024-04-23T18:08:36.412 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:36.609 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:36 smithi026 ceph-mon[29364]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:36.609 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:36.609 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:36 smithi026 ceph-mon[29364]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:08:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 -- 172.21.15.26:0/2328405859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 msgr2=0x7f8590103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 --2- 172.21.15.26:0/2328405859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590103ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8584009910 tx=0x7f8584009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 -- 172.21.15.26:0/2328405859 shutdown_connections 2024-04-23T18:08:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 --2- 172.21.15.26:0/2328405859 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590103ae0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 -- 172.21.15.26:0/2328405859 >> 172.21.15.26:0/2328405859 conn(0x7f85900fce90 msgr2=0x7f85900ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 -- 172.21.15.26:0/2328405859 shutdown_connections 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.818+0000 7f8596ec8700 1 -- 172.21.15.26:0/2328405859 wait complete. 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8596ec8700 1 Processor -- start 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8596ec8700 1 -- start start 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8596ec8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8596ec8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f859010c9d0 con 0x7f8590103700 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8594c64700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8594c64700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48216/0 (socket says 172.21.15.26:48216) 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.819+0000 7f8594c64700 1 -- 172.21.15.26:0/50857018 learned_addr learned my addr 172.21.15.26:0/50857018 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f8594c64700 1 -- 172.21.15.26:0/50857018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8590103250 con 0x7f8590103700 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f8594c64700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8584006010 tx=0x7f8584005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8584026070 con 0x7f8590103700 2024-04-23T18:08:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f858400bc00 con 0x7f8590103700 2024-04-23T18:08:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85901018a0 con 0x7f8590103700 2024-04-23T18:08:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8584003910 con 0x7f8590103700 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.820+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8590101d60 con 0x7f8590103700 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.821+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8584029070 con 0x7f8590103700 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.821+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f859006b0b0 con 0x7f8590103700 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.821+0000 7f858dffb700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 0x7f858003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.821+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f858401e070 con 0x7f8590103700 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.821+0000 7f858ffff700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 0x7f858003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:37.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.822+0000 7f858ffff700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 0x7f858003bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f857c006fd0 tx=0x7f857c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:37.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:37.826+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8584013070 con 0x7f8590103700 2024-04-23T18:08:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:38.039+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}) -- 0x7f8590102900 con 0x7f85800396b0 2024-04-23T18:08:38.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:38 smithi026 ceph-mon[29364]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:39 smithi026 ceph-mon[29364]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:08:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:40 smithi026 ceph-mon[29364]: Deploying cephadm binary to smithi131 2024-04-23T18:08:40.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:40 smithi026 ceph-mon[29364]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:40.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:40 smithi026 ceph-mon[29364]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:08:40.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:40 smithi026 ceph-mon[29364]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:40.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:40 smithi026 ceph-mon[29364]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:08:42.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.383+0000 7f858dffb700 1 -- 172.21.15.26:0/50857018 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8590102900 con 0x7f85800396b0 2024-04-23T18:08:42.378 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi131' with addr '172.21.15.131' 2024-04-23T18:08:42.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 msgr2=0x7f858003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 0x7f858003bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f857c006fd0 tx=0x7f857c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:42.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 msgr2=0x7f8590101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8584006010 tx=0x7f8584005780 comp rx=0 tx=0).stop 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 shutdown_connections 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f85800396b0 0x7f858003bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 --2- 172.21.15.26:0/50857018 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8590103700 0x7f8590101360 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 >> 172.21.15.26:0/50857018 conn(0x7f85900fce90 msgr2=0x7f85901044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 shutdown_connections 2024-04-23T18:08:42.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:42.386+0000 7f8596ec8700 1 -- 172.21.15.26:0/50857018 wait complete. 2024-04-23T18:08:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:43.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:08:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:43.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:43 smithi026 ceph-mon[29364]: Added host smithi131 2024-04-23T18:08:43.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:43.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:44.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:44 smithi026 ceph-mon[29364]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:44.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- 172.21.15.26:0/1921229344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 msgr2=0x7f13241008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 --2- 172.21.15.26:0/1921229344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f13241008b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1320009910 tx=0x7f1320009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- 172.21.15.26:0/1921229344 shutdown_connections 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 --2- 172.21.15.26:0/1921229344 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f13241008b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- 172.21.15.26:0/1921229344 >> 172.21.15.26:0/1921229344 conn(0x7f13240fc5f0 msgr2=0x7f13240fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- 172.21.15.26:0/1921229344 shutdown_connections 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- 172.21.15.26:0/1921229344 wait complete. 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 Processor -- start 2024-04-23T18:08:44.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.841+0000 7f132b709700 1 -- start start 2024-04-23T18:08:44.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f132b709700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f132b709700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f132410eba0 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13294a5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13294a5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34764/0 (socket says 172.21.15.26:34764) 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13294a5700 1 -- 172.21.15.26:0/465695489 learned_addr learned my addr 172.21.15.26:0/465695489 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13294a5700 1 -- 172.21.15.26:0/465695489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132419a610 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13294a5700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13200056d0 tx=0x7f13200057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1320026070 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1320004370 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132419a8a0 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13200039b0 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.842+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132419e9a0 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.843+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1320029070 con 0x7f13241033a0 2024-04-23T18:08:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.843+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1324101ad0 con 0x7f13241033a0 2024-04-23T18:08:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.843+0000 7f13167fc700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 0x7f131003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.844+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f132001e070 con 0x7f13241033a0 2024-04-23T18:08:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.844+0000 7f1328ca4700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 0x7f131003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:44.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.844+0000 7f1328ca4700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 0x7f131003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1318006fd0 tx=0x7f1318006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:44.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:44.849+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1320013070 con 0x7f13241033a0 2024-04-23T18:08:45.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.068+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13240640a0 con 0x7f13100396b0 2024-04-23T18:08:45.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.069+0000 7f13167fc700 1 -- 172.21.15.26:0/465695489 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f13240640a0 con 0x7f13100396b0 2024-04-23T18:08:45.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:08:45.063 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}] 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 msgr2=0x7f131003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 0x7f131003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1318006fd0 tx=0x7f1318006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 msgr2=0x7f132419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13200056d0 tx=0x7f13200057b0 comp rx=0 tx=0).stop 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 shutdown_connections 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f13100396b0 0x7f131003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 --2- 172.21.15.26:0/465695489 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f13241033a0 0x7f132419a0d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 >> 172.21.15.26:0/465695489 conn(0x7f13240fc5f0 msgr2=0x7f13240fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 shutdown_connections 2024-04-23T18:08:45.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:45.071+0000 7f132b709700 1 -- 172.21.15.26:0/465695489 wait complete. 2024-04-23T18:08:45.653 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T18:08:45.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T18:08:45.894 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:08:46.286 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:45 smithi026 ceph-mon[29364]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:46.287 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:45 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:46.287 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:45 smithi026 ceph-mon[29364]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:08:47.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 -- 172.21.15.26:0/1254872575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 msgr2=0x7f5d941088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 --2- 172.21.15.26:0/1254872575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d941088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400b0a0 tx=0x7f5d8400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 -- 172.21.15.26:0/1254872575 shutdown_connections 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 --2- 172.21.15.26:0/1254872575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d941088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 -- 172.21.15.26:0/1254872575 >> 172.21.15.26:0/1254872575 conn(0x7f5d94076ea0 msgr2=0x7f5d940772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 -- 172.21.15.26:0/1254872575 shutdown_connections 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.489+0000 7f5d98a32700 1 -- 172.21.15.26:0/1254872575 wait complete. 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d98a32700 1 Processor -- start 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d98a32700 1 -- start start 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d98a32700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d98a32700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9410c920 con 0x7f5d941084e0 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d9259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d9259c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34778/0 (socket says 172.21.15.26:34778) 2024-04-23T18:08:47.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d9259c700 1 -- 172.21.15.26:0/3893289217 learned_addr learned my addr 172.21.15.26:0/3893289217 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:08:47.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.490+0000 7f5d9259c700 1 -- 172.21.15.26:0/3893289217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d941983b0 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d9259c700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006130 tx=0x7f5d840097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d84021020 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d840079b0 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d94198640 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d84017040 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.491+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9419c6f0 con 0x7f5d941084e0 2024-04-23T18:08:47.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.492+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5d84024020 con 0x7f5d941084e0 2024-04-23T18:08:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.492+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d94103b90 con 0x7f5d941084e0 2024-04-23T18:08:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.492+0000 7f5d8b7fe700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 0x7f5d8003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.492+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d8400f050 con 0x7f5d941084e0 2024-04-23T18:08:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.492+0000 7f5d91d9b700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 0x7f5d8003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.493+0000 7f5d91d9b700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 0x7f5d8003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c006fd0 tx=0x7f5d7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:47.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.497+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d8402b020 con 0x7f5d941084e0 2024-04-23T18:08:47.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.711+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5d940640a0 con 0x7f5d941084e0 2024-04-23T18:08:47.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.990+0000 7f5d8b7fe700 1 -- 172.21.15.26:0/3893289217 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5d8401b7b0 con 0x7f5d941084e0 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 msgr2=0x7f5d8003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 0x7f5d8003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c006fd0 tx=0x7f5d7c006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 msgr2=0x7f5d94197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006130 tx=0x7f5d840097d0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 shutdown_connections 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5d800396b0 0x7f5d8003bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 --2- 172.21.15.26:0/3893289217 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d941084e0 0x7f5d94197e70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 >> 172.21.15.26:0/3893289217 conn(0x7f5d94076ea0 msgr2=0x7f5d940fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 shutdown_connections 2024-04-23T18:08:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:08:47.992+0000 7f5d98a32700 1 -- 172.21.15.26:0/3893289217 wait complete. 2024-04-23T18:08:47.989 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-04-23T18:08:48.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:47 smithi026 ceph-mon[29364]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:48.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:47 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3893289217' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:08:48.650 INFO:tasks.cephadm:Adding mon.a on smithi026 2024-04-23T18:08:48.651 INFO:tasks.cephadm:Adding mon.b on smithi110 2024-04-23T18:08:48.651 INFO:tasks.cephadm:Adding mon.c on smithi131 2024-04-23T18:08:48.651 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c' 2024-04-23T18:08:48.889 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:08:49.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3893289217' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:08:49.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:48 smithi026 ceph-mon[29364]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:50.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 -- 172.21.15.131:0/1151492137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 msgr2=0x7f23981088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:50.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 --2- 172.21.15.131:0/1151492137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f23981088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2394009910 tx=0x7f2394009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:50.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 -- 172.21.15.131:0/1151492137 shutdown_connections 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 --2- 172.21.15.131:0/1151492137 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f23981088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 -- 172.21.15.131:0/1151492137 >> 172.21.15.131:0/1151492137 conn(0x7f23980fdf80 msgr2=0x7f23981003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 -- 172.21.15.131:0/1151492137 shutdown_connections 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.344+0000 7f23a0367700 1 -- 172.21.15.131:0/1151492137 wait complete. 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f23a0367700 1 Processor -- start 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f23a0367700 1 -- start start 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f23a0367700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f23a0367700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f239810c940 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f239e103700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f239e103700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.131:52460/0 (socket says 172.21.15.131:52460) 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.345+0000 7f239e103700 1 -- 172.21.15.131:0/1394362023 learned_addr learned my addr 172.21.15.131:0/1394362023 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f239e103700 1 -- 172.21.15.131:0/1394362023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2398076c30 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f239e103700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f23940056d0 tx=0x7f23940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2394026070 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23980752e0 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f239400bc00 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2394003920 con 0x7f23981084e0 2024-04-23T18:08:50.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.346+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23980757a0 con 0x7f23981084e0 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.347+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2398103b90 con 0x7f23981084e0 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.347+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2394029070 con 0x7f23981084e0 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.347+0000 7f238f7fe700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 0x7f238403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.347+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f239401e070 con 0x7f23981084e0 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.347+0000 7f239d902700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 0x7f238403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:50.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.348+0000 7f239d902700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 0x7f238403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2388006fd0 tx=0x7f2388006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:50.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.352+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2394013070 con 0x7f23981084e0 2024-04-23T18:08:50.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:50 smithi026 ceph-mon[29364]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:50.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:50 smithi026 ceph-mon[29364]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:08:50.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:50.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:50.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:50.553 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.558+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}) -- 0x7f23980762e0 con 0x7f2384039650 2024-04-23T18:08:50.560 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.565+0000 7f238f7fe700 1 -- 172.21.15.131:0/1394362023 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f23980762e0 con 0x7f2384039650 2024-04-23T18:08:50.560 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mon update... 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 msgr2=0x7f238403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 0x7f238403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2388006fd0 tx=0x7f2388006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 msgr2=0x7f2398074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f23940056d0 tx=0x7f23940057b0 comp rx=0 tx=0).stop 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 shutdown_connections 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2384039650 0x7f238403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 --2- 172.21.15.131:0/1394362023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f23981084e0 0x7f2398074d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:50.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.567+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 >> 172.21.15.131:0/1394362023 conn(0x7f23980fdf80 msgr2=0x7f239810a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:50.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.568+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 shutdown_connections 2024-04-23T18:08:50.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:50.568+0000 7f23a0367700 1 -- 172.21.15.131:0/1394362023 wait complete. 2024-04-23T18:08:51.177 DEBUG:teuthology.orchestra.run.smithi110:mon.b> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T18:08:51.180 DEBUG:teuthology.orchestra.run.smithi131:mon.c> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T18:08:51.182 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:08:51.182 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c;count:3 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:08:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:08:51.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:08:51.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:51.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:51 smithi026 ceph-mon[29364]: Deploying daemon mon.c on smithi131 2024-04-23T18:08:52.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:52 smithi026 ceph-mon[29364]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:53.134 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:08:54.957 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1965282547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 msgr2=0x7ff8b4108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1965282547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4108a10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009910 tx=0x7ff8a4009c20 comp rx=0 tx=0).stop 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1965282547 shutdown_connections 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1965282547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4108a10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1965282547 >> 172.21.15.131:0/1965282547 conn(0x7ff8b40fe050 msgr2=0x7ff8b4100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1965282547 shutdown_connections 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1965282547 wait complete. 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 Processor -- start 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.961+0000 7ff8bc20a700 1 -- start start 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8bc20a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8bc20a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410caa0 con 0x7ff8b4108630 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8b9fa6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8b9fa6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.131:52478/0 (socket says 172.21.15.131:52478) 2024-04-23T18:08:54.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8b9fa6700 1 -- 172.21.15.131:0/1112435553 learned_addr learned my addr 172.21.15.131:0/1112435553 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8b9fa6700 1 -- 172.21.15.131:0/1112435553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b41984d0 con 0x7ff8b4108630 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.962+0000 7ff8b9fa6700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40056d0 tx=0x7ff8a40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.963+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8a4026070 con 0x7ff8b4108630 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.963+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a400bc00 con 0x7ff8b4108630 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.963+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b4198760 con 0x7ff8b4108630 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.963+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8a4003920 con 0x7ff8b4108630 2024-04-23T18:08:54.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.963+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b419c860 con 0x7ff8b4108630 2024-04-23T18:08:54.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:08:54 smithi026 ceph-mon[29364]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff8a4029070 con 0x7ff8b4108630 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b4103ca0 con 0x7ff8b4108630 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8aaffd700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 0x7ff8a003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff8a401e070 con 0x7ff8b4108630 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8b97a5700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 0x7ff8a003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:54.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.964+0000 7ff8b97a5700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 0x7ff8a003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0006fd0 tx=0x7ff8b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:54.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:54.970+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8a4013070 con 0x7ff8b4108630 2024-04-23T18:08:55.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.258+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff8b41992f0 con 0x7ff8b4108630 2024-04-23T18:08:55.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.259+0000 7ff8aaffd700 1 -- 172.21.15.131:0/1112435553 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ff8a4030090 con 0x7ff8b4108630 2024-04-23T18:08:55.254 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:08:55.254 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":1,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:07:10.862733Z","created":"2024-04-23T18:07:10.862733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T18:08:55.255 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 msgr2=0x7ff8a003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 0x7ff8a003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0006fd0 tx=0x7ff8b0006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 msgr2=0x7ff8b4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40056d0 tx=0x7ff8a40057b0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 shutdown_connections 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff8a0039650 0x7ff8a003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 --2- 172.21.15.131:0/1112435553 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8b4108630 0x7ff8b4197f90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 >> 172.21.15.131:0/1112435553 conn(0x7ff8b40fe050 msgr2=0x7ff8b40ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 shutdown_connections 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:08:55.261+0000 7ff8bc20a700 1 -- 172.21.15.131:0/1112435553 wait complete. 2024-04-23T18:08:55.256 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 1 2024-04-23T18:08:57.082 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:08:57.082 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:08:57.328 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: mon.a calling monitor election 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:00.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: mon.c calling monitor election 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: monmap epoch 2 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: last_changed 2024-04-23T18:08:55.310321+0000 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: min_mon_release 19 (squid) 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: election_strategy: 1 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:00.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: fsmap 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: mgrmap e13: x(active, since 49s) 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: overall HEALTH_OK 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:00.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:01.615 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 -- 172.21.15.131:0/211674967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 msgr2=0x7f9300005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:01.615 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 --2- 172.21.15.131:0/211674967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9300005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9314102490 tx=0x7f931000feb0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.615 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 -- 172.21.15.131:0/211674967 shutdown_connections 2024-04-23T18:09:01.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 --2- 172.21.15.131:0/211674967 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9300005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 -- 172.21.15.131:0/211674967 >> 172.21.15.131:0/211674967 conn(0x7f93140fe050 msgr2=0x7f9314100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:01.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.620+0000 7f931b35c700 1 -- 172.21.15.131:0/211674967 shutdown_connections 2024-04-23T18:09:01.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.621+0000 7f931b35c700 1 -- 172.21.15.131:0/211674967 wait complete. 2024-04-23T18:09:01.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.621+0000 7f931b35c700 1 Processor -- start 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f931b35c700 1 -- start start 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f931b35c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f931b35c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f931b35c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f931410a4b0 con 0x7f93000035c0 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f931b35c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f931410a330 con 0x7f93141984d0 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.131:45892/0 (socket says 172.21.15.131:45892) 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 -- 172.21.15.131:0/1852486862 learned_addr learned my addr 172.21.15.131:0/1852486862 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:36748/0 (socket says 172.21.15.131:36748) 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 -- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 msgr2=0x7f931419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 -- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 msgr2=0x7f931419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:09:01.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93188f7700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 -- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 msgr2=0x7f931419c860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.622+0000 7f93190f8700 1 -- 172.21.15.131:0/1852486862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9314198b10 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f93190f8700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9310010010 tx=0x7f93100050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9310013960 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93100044e0 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931419cda0 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9310011c70 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.623+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f931419d380 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.624+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9310023070 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.624+0000 7f930a7fc700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 0x7f92f803b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.624+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9310020050 con 0x7f93000035c0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.624+0000 7f93188f7700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 0x7f92f803b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:01.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.624+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9314103ca0 con 0x7f93000035c0 2024-04-23T18:09:01.621 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.625+0000 7f93188f7700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 0x7f92f803b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9304009850 tx=0x7f9304008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:01.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.630+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9310028020 con 0x7f93000035c0 2024-04-23T18:09:01.625 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:01 smithi131 ceph-mon[33041]: Deploying daemon mon.b on smithi110 2024-04-23T18:09:01.625 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:01 smithi026 ceph-mon[29364]: Deploying daemon mon.b on smithi110 2024-04-23T18:09:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:01.904 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.910+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f93141992f0 con 0x7f93000035c0 2024-04-23T18:09:01.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.910+0000 7f930a7fc700 1 -- 172.21.15.131:0/1852486862 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f931001a390 con 0x7f93000035c0 2024-04-23T18:09:01.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:09:01.905 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":2,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:08:55.310321Z","created":"2024-04-23T18:07:10.862733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 msgr2=0x7f92f803b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 0x7f92f803b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9304009850 tx=0x7f9304008040 comp rx=0 tx=0).stop 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 msgr2=0x7f9314197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9310010010 tx=0x7f93100050d0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 shutdown_connections 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f92f80393f0 0x7f92f803b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93000035c0 0x7f9314197f90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 --2- 172.21.15.131:0/1852486862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93141984d0 0x7f931419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 >> 172.21.15.131:0/1852486862 conn(0x7f93140fe050 msgr2=0x7f93140fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 shutdown_connections 2024-04-23T18:09:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:01.912+0000 7f931b35c700 1 -- 172.21.15.131:0/1852486862 wait complete. 2024-04-23T18:09:01.908 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 2 2024-04-23T18:09:02.466 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:02 smithi131 ceph-mon[33041]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:02.466 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:02 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/1852486862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:02 smithi026 ceph-mon[29364]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:02.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/1852486862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:03.495 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:09:03.495 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:09:03.737 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:09:03.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:03 smithi110 podman[32567]: 2024-04-23 18:09:03.786575733 +0000 UTC m=+0.018405089 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:09:04.139 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 podman[32567]: 2024-04-23 18:09:04.094471081 +0000 UTC m=+0.326300408 container init cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 podman[32567]: 2024-04-23 18:09:04.097942309 +0000 UTC m=+0.329771638 container start cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pidfile_write: ignore empty --pid-file 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: load: jerasure load: lrc 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Git sha 0 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: DB SUMMARY 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: DB Session ID: L3EQTZ31Y38IC87YH03R 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:09:04.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.env: 0x5595313b9cc0 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:09:04.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.info_log: 0x5595332f9be0 2024-04-23T18:09:04.142 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.statistics: (nil) 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.use_fsync: 0 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:09:04.144 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.db_log_dir: 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.wal_dir: 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:09:04.145 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.write_buffer_manager: 0x55953328d0e0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:09:04.146 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.unordered_write: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.row_cache: None 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.wal_filter: None 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.wal_compression: 0 2024-04-23T18:09:04.147 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:09:04.148 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_open_files: -1 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:09:04.149 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Compression algorithms supported: 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kZSTD supported: 0 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-23T18:09:04.150 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.merge_operator: 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_filter: None 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5595332f9dc0) 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:09:04.151 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_type: 0 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: data_block_index_type: 0 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_shortening: 1 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: checksum: 4 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: no_block_cache: 0 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache: 0x5595332bf1f0 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_options: 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: capacity : 536870912 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: num_shard_bits : 4 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: strict_capacity_limit : 0 2024-04-23T18:09:04.152 INFO:journalctl@ceph.mon.b.smithi110.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_compressed: (nil) 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: persistent_cache: (nil) 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_size: 4096 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_size_deviation: 10 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_restart_interval: 16 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_block_restart_interval: 1 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: metadata_block_size: 4096 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: partition_filters: 0 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: use_delta_encoding: 1 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: filter_policy: bloomfilter 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: whole_key_filtering: 1 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: verify_compression: 0 2024-04-23T18:09:04.153 INFO:journalctl@ceph.mon.b.smithi110.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: format_version: 5 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: enable_index_compression: 1 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_align: 0 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: max_auto_readahead_size: 262144 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: prepopulate_block_cache: 0 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression: NoCompression 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:09:04.154 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.num_levels: 7 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:09:04.155 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:09:04.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:09:04.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:09:04.158 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:09:04.159 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.ttl: 2592000 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:09:04.160 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:09:04.161 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8f54d9c9-e185-483e-aee4-1f7dd15c3d8f 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895744127376, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895744127793, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895744, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8f54d9c9-e185-483e-aee4-1f7dd15c3d8f", "db_session_id": "L3EQTZ31Y38IC87YH03R", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895744127873, "job": 1, "event": "recovery_finished"} 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:09:04.162 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5595333d4000 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: DB pointer 0x5595333be000 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** DB Stats ** 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:09:04.163 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** Compaction Stats [default] ** 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.164 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** Compaction Stats [default] ** 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:09:04.165 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: Block cache BinnedLRUCache@0x5595332bf1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-23T18:09:04.166 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: using public_addr v2:172.21.15.110:0/0 -> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: starting mon.b rank -1 at public addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] at bind addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(???) e0 preinit fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).mds e1 new map 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: e1 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: legacy client fscid: -1 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout: No filesystems configured 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T18:09:04.167 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Added host smithi110 2024-04-23T18:09:04.168 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.169 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Deploying cephadm binary to smithi131 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.170 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Added host smithi131 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:09:04.171 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3893289217' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3893289217' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:09:04.172 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c;count:3 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:04.173 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Deploying daemon mon.c on smithi131 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.a calling monitor election 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.c calling monitor election 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:09:04.175 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: monmap epoch 2 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: last_changed 2024-04-23T18:08:55.310321+0000 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: min_mon_release 19 (squid) 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: election_strategy: 1 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: fsmap 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mgrmap e13: x(active, since 49s) 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: overall HEALTH_OK 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.176 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: Deploying daemon mon.b on smithi110 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/1852486862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:04.177 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 ceph-mon[32585]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T18:09:04.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 bash[32567]: cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 2024-04-23T18:09:04.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:04 smithi110 systemd[1]: Started Ceph mon.b for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 -- 172.21.15.131:0/3271656804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33100057d0 msgr2=0x7f3310007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 --2- 172.21.15.131:0/3271656804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33100057d0 0x7f3310007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f33281013a0 tx=0x7f331c00fe60 comp rx=0 tx=0).stop 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 -- 172.21.15.131:0/3271656804 shutdown_connections 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 --2- 172.21.15.131:0/3271656804 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f33100057d0 0x7f3310007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 -- 172.21.15.131:0/3271656804 >> 172.21.15.131:0/3271656804 conn(0x7f33280fcf80 msgr2=0x7f33280ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:09.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 -- 172.21.15.131:0/3271656804 shutdown_connections 2024-04-23T18:09:09.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.443+0000 7f332d86c700 1 -- 172.21.15.131:0/3271656804 wait complete. 2024-04-23T18:09:09.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 Processor -- start 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 -- start start 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 0x7f332819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f33281082e0 con 0x7f3328196460 2024-04-23T18:09:09.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f332806bb90 con 0x7f33100057d0 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f332d86c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3328108460 con 0x7f332819ad30 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f33277fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f33277fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:55796/0 (socket says 172.21.15.131:55796) 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f33277fe700 1 -- 172.21.15.131:0/2070948932 learned_addr learned my addr 172.21.15.131:0/2070948932 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.444+0000 7f33267fc700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 0x7f332819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f3326ffd700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33277fe700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 msgr2=0x7f332819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33277fe700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 msgr2=0x7f332819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33277fe700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f3326ffd700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 msgr2=0x7f3328195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-23T18:09:09.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f3326ffd700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 msgr2=0x7f3328195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:09:09.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f3326ffd700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:09:09.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33277fe700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:09.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f3326ffd700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:09.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 msgr2=0x7f332819b1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:09.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 msgr2=0x7f3328195f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 -- 172.21.15.131:0/2070948932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33281a2aa0 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f33267fc700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 0x7f332819a7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f331800b550 tx=0x7f331800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318004230 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33180043d0 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318011b90 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33281a2cd0 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.445+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33281a3200 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f331801a400 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f332c86a700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 0x7f331403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33280640a0 con 0x7f3328196460 2024-04-23T18:09:09.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f331804c650 con 0x7f3328196460 2024-04-23T18:09:09.445 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f3326ffd700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 0x7f331403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.445 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.447+0000 7f3326ffd700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 0x7f331403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f331c00fa10 tx=0x7f331c005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:09.449 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.455+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33180151c0 con 0x7f3328196460 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: mon.a calling monitor election 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: mon.c calling monitor election 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: monmap epoch 3 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: min_mon_release 19 (squid) 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: election_strategy: 1 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: fsmap 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: mgrmap e13: x(active, since 58s) 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: overall HEALTH_OK 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: mon.a calling monitor election 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: mon.c calling monitor election 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: monmap epoch 3 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: min_mon_release 19 (squid) 2024-04-23T18:09:09.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: election_strategy: 1 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: fsmap 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: mgrmap e13: x(active, since 58s) 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: overall HEALTH_OK 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:09.739 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.744+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f332802e140 con 0x7f3328196460 2024-04-23T18:09:09.739 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.745+0000 7f332c86a700 1 -- 172.21.15.131:0/2070948932 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f3318018510 con 0x7f3328196460 2024-04-23T18:09:09.740 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-04-23T18:09:09.740 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":3,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:09:04.150799Z","created":"2024-04-23T18:07:10.862733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.747+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 msgr2=0x7f331403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.747+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 0x7f331403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f331c00fa10 tx=0x7f331c005de0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.747+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 msgr2=0x7f332819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.747+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 0x7f332819a7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f331800b550 tx=0x7f331800b910 comp rx=0 tx=0).stop 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 shutdown_connections 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3314039ae0 0x7f331403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f332819ad30 0x7f332819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100057d0 0x7f3328195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 --2- 172.21.15.131:0/2070948932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3328196460 0x7f332819a7f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 >> 172.21.15.131:0/2070948932 conn(0x7f33280fcf80 msgr2=0x7f3328106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:09.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 shutdown_connections 2024-04-23T18:09:09.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:09.748+0000 7f332d86c700 1 -- 172.21.15.131:0/2070948932 wait complete. 2024-04-23T18:09:09.743 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 3 2024-04-23T18:09:10.304 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:10 smithi131 ceph-mon[33041]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:10.305 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:10.305 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:10.305 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:10 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/2070948932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:10.305 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:10.331 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T18:09:10.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T18:09:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:10 smithi026 ceph-mon[29364]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:10 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/2070948932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:11.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: mon.a calling monitor election 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: mon.c calling monitor election 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: monmap epoch 3 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:11.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: min_mon_release 19 (squid) 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: election_strategy: 1 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: fsmap 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: mgrmap e13: x(active, since 58s) 2024-04-23T18:09:11.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: overall HEALTH_OK 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/2070948932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:09:11.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:12.298 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mon.b calling monitor election 2024-04-23T18:09:12.298 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:12.298 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:09:12.298 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mon.b calling monitor election 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mon.a calling monitor election 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mon.c calling monitor election 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: monmap epoch 3 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: min_mon_release 19 (squid) 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: election_strategy: 1 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:12.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: fsmap 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: mgrmap e13: x(active, since 60s) 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: overall HEALTH_OK 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:12.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 -- 172.21.15.26:0/2834037652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 msgr2=0x7f9e4c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 --2- 172.21.15.26:0/2834037652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c10aa40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009910 tx=0x7f9e3c009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 -- 172.21.15.26:0/2834037652 shutdown_connections 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 --2- 172.21.15.26:0/2834037652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c10aa40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 -- 172.21.15.26:0/2834037652 >> 172.21.15.26:0/2834037652 conn(0x7f9e4c100190 msgr2=0x7f9e4c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 -- 172.21.15.26:0/2834037652 shutdown_connections 2024-04-23T18:09:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.333+0000 7f9e5084a700 1 -- 172.21.15.26:0/2834037652 wait complete. 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 Processor -- start 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 -- start start 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 0x7f9e4c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e4c19f130 0x7f9e4c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c10eb50 con 0x7f9e4c10a660 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c10e9d0 con 0x7f9e4c19f130 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e5084a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c10ecd0 con 0x7f9e4c19a860 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.334+0000 7f9e4a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59552/0 (socket says 172.21.15.26:59552) 2024-04-23T18:09:12.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 -- 172.21.15.26:0/3379797702 learned_addr learned my addr 172.21.15.26:0/3379797702 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e49d9b700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 0x7f9e4c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4ad9d700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e4c19f130 0x7f9e4c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 -- 172.21.15.26:0/3379797702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 msgr2=0x7f9e4c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 0x7f9e4c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 -- 172.21.15.26:0/3379797702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e4c19f130 msgr2=0x7f9e4c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e4c19f130 0x7f9e4c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 -- 172.21.15.26:0/3379797702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c1a6ea0 con 0x7f9e4c10a660 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e49d9b700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 0x7f9e4c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e4a59c700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005b40 tx=0x7f9e3c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c026070 con 0x7f9e4c10a660 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e3c00b6f0 con 0x7f9e4c10a660 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c009e60 con 0x7f9e4c10a660 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.335+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4c1a70d0 con 0x7f9e4c10a660 2024-04-23T18:09:12.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.336+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4c1a75c0 con 0x7f9e4c10a660 2024-04-23T18:09:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.336+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9e3c029070 con 0x7f9e4c10a660 2024-04-23T18:09:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.337+0000 7f9e3b7fe700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 0x7f9e3403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.337+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9e3c04d450 con 0x7f9e4c10a660 2024-04-23T18:09:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.337+0000 7f9e49d9b700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 0x7f9e3403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.337+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e4c0640a0 con 0x7f9e4c10a660 2024-04-23T18:09:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.337+0000 7f9e49d9b700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 0x7f9e3403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009ec0 tx=0x7f9e40009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:12.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.342+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e3c030020 con 0x7f9e4c10a660 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mon.b calling monitor election 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:09:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mon.b calling monitor election 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mon.a calling monitor election 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mon.c calling monitor election 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: monmap epoch 3 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: min_mon_release 19 (squid) 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: election_strategy: 1 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: fsmap 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:12.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: mgrmap e13: x(active, since 60s) 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: overall HEALTH_OK 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:12.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:12.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mon.b calling monitor election 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mon.b calling monitor election 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mon.a calling monitor election 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mon.c calling monitor election 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:09:12.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: monmap epoch 3 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: last_changed 2024-04-23T18:09:04.150799+0000 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: created 2024-04-23T18:07:10.862733+0000 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: min_mon_release 19 (squid) 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: election_strategy: 1 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: 0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: 1: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.c 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: fsmap 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: mgrmap e13: x(active, since 60s) 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: overall HEALTH_OK 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:12.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.557+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9e4c1a7820 con 0x7f9e4c10a660 2024-04-23T18:09:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.557+0000 7f9e3b7fe700 1 -- 172.21.15.26:0/3379797702 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f9e3c013070 con 0x7f9e4c10a660 2024-04-23T18:09:12.552 INFO:teuthology.orchestra.run.smithi026.stdout:# minimal ceph.conf for 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.552 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-04-23T18:09:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.552 INFO:teuthology.orchestra.run.smithi026.stdout: mon_host = [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.560+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 msgr2=0x7f9e3403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.560+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 0x7f9e3403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009ec0 tx=0x7f9e40009210 comp rx=0 tx=0).stop 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.560+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 msgr2=0x7f9e4c19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.560+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005b40 tx=0x7f9e3c004f80 comp rx=0 tx=0).stop 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 shutdown_connections 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9e34039ae0 0x7f9e3403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e4c19f130 0x7f9e4c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9e4c10a660 0x7f9e4c19a320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 --2- 172.21.15.26:0/3379797702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e4c19a860 0x7f9e4c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 >> 172.21.15.26:0/3379797702 conn(0x7f9e4c100190 msgr2=0x7f9e4c10c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 shutdown_connections 2024-04-23T18:09:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:12.561+0000 7f9e5084a700 1 -- 172.21.15.26:0/3379797702 wait complete. 2024-04-23T18:09:13.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:13 smithi026 ceph-mon[29364]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:09:13.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:13 smithi026 ceph-mon[29364]: Reconfiguring daemon mon.a on smithi026 2024-04-23T18:09:13.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:13.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3379797702' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:13.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:13 smithi110 ceph-mon[32585]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:09:13.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:13 smithi110 ceph-mon[32585]: Reconfiguring daemon mon.a on smithi026 2024-04-23T18:09:13.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:13.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3379797702' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:13.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:13 smithi131 ceph-mon[33041]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:09:13.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:13 smithi131 ceph-mon[33041]: Reconfiguring daemon mon.a on smithi026 2024-04-23T18:09:13.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:09:13.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3379797702' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:13.623 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T18:09:13.623 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:09:13.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:09:13.650 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:09:13.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:13.716 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:09:13.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:09:13.746 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:09:13.747 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:13.813 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:09:13.813 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:09:13.845 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:09:13.845 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:13.913 INFO:tasks.cephadm:Adding mgr.x on smithi026 2024-04-23T18:09:13.913 INFO:tasks.cephadm:Adding mgr.y on smithi110 2024-04-23T18:09:13.913 INFO:tasks.cephadm:Adding mgr.z on smithi131 2024-04-23T18:09:13.913 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi026=x;smithi110=y;smithi131=z' 2024-04-23T18:09:14.188 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:09:14.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:14 smithi131 ceph-mon[33041]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:14.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:14 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:14.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:14 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:14.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:14 smithi110 ceph-mon[32585]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:14.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:14 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:14.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:14 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:14.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:14 smithi026 ceph-mon[29364]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:14.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:14 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:14.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:14 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:15.343 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:15 smithi131 ceph-mon[33041]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:09:15.343 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:15.343 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:15.343 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:15.343 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:15 smithi131 ceph-mon[33041]: Reconfiguring daemon mon.b on smithi110 2024-04-23T18:09:15.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:15 smithi110 ceph-mon[32585]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:09:15.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:15.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:15.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:15.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:15 smithi110 ceph-mon[32585]: Reconfiguring daemon mon.b on smithi110 2024-04-23T18:09:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:15 smithi026 ceph-mon[29364]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:09:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:15 smithi026 ceph-mon[29364]: Reconfiguring daemon mon.b on smithi110 2024-04-23T18:09:15.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 -- 172.21.15.131:0/3984231353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 msgr2=0x7fb3981018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:15.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 --2- 172.21.15.131:0/3984231353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3981018e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009910 tx=0x7fb38c009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 -- 172.21.15.131:0/3984231353 shutdown_connections 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 --2- 172.21.15.131:0/3984231353 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3981018e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 -- 172.21.15.131:0/3984231353 >> 172.21.15.131:0/3984231353 conn(0x7fb3980fca80 msgr2=0x7fb3980feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 -- 172.21.15.131:0/3984231353 shutdown_connections 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 -- 172.21.15.131:0/3984231353 wait complete. 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.650+0000 7fb39d304700 1 Processor -- start 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 -- start start 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3980fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3980fd020 0x7fb3980fd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb398107400 con 0x7fb3981014c0 2024-04-23T18:09:15.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb398107280 con 0x7fb3980fd020 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb39d304700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb398107580 con 0x7fb3980fd9e0 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:55810/0 (socket says 172.21.15.131:55810) 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 -- 172.21.15.131:0/200645875 learned_addr learned my addr 172.21.15.131:0/200645875 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3967fc700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3980fd020 0x7fb3980fd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb396ffd700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3980fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 -- 172.21.15.131:0/200645875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3980fd020 msgr2=0x7fb3980fd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3980fd020 0x7fb3980fd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 -- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 msgr2=0x7fb3980fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3980fe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 -- 172.21.15.131:0/200645875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3981a2330 con 0x7fb3980fd9e0 2024-04-23T18:09:15.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.651+0000 7fb3977fe700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb39000ba50 tx=0x7fb39000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:15.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.652+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390019070 con 0x7fb3980fd9e0 2024-04-23T18:09:15.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.652+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb390014440 con 0x7fb3980fd9e0 2024-04-23T18:09:15.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.652+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3981a2620 con 0x7fb3980fd9e0 2024-04-23T18:09:15.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.652+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39000ce60 con 0x7fb3980fd9e0 2024-04-23T18:09:15.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.652+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3981a2b90 con 0x7fb3980fd9e0 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.653+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb39001c070 con 0x7fb3980fd9e0 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.653+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3980640a0 con 0x7fb3980fd9e0 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.653+0000 7fb37ffff700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 0x7fb38003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.653+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb39004be60 con 0x7fb3980fd9e0 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.653+0000 7fb396ffd700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 0x7fb38003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:15.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.654+0000 7fb396ffd700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 0x7fb38003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b5c0 tx=0x7fb38c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:15.652 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.658+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3900145f0 con 0x7fb3980fd9e0 2024-04-23T18:09:15.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.878+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi026=x;smithi110=y;smithi131=z", "target": ["mon-mgr", ""]}) -- 0x7fb398062e80 con 0x7fb380039ae0 2024-04-23T18:09:15.878 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.884+0000 7fb37ffff700 1 -- 172.21.15.131:0/200645875 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb398062e80 con 0x7fb380039ae0 2024-04-23T18:09:15.878 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mgr update... 2024-04-23T18:09:15.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 msgr2=0x7fb38003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 0x7fb38003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b5c0 tx=0x7fb38c01a040 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 msgr2=0x7fb3981a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb39000ba50 tx=0x7fb39000bd60 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 shutdown_connections 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb380039ae0 0x7fb38003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb3980fd9e0 0x7fb3981a1d70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb3981014c0 0x7fb3980fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 --2- 172.21.15.131:0/200645875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3980fd020 0x7fb3980fd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 >> 172.21.15.131:0/200645875 conn(0x7fb3980fca80 msgr2=0x7fb39806b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 shutdown_connections 2024-04-23T18:09:15.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:09:15.887+0000 7fb39d304700 1 -- 172.21.15.131:0/200645875 wait complete. 2024-04-23T18:09:16.563 DEBUG:teuthology.orchestra.run.smithi110:mgr.y> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T18:09:16.566 DEBUG:teuthology.orchestra.run.smithi131:mgr.z> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.z.service 2024-04-23T18:09:16.569 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T18:09:16.569 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:09:16.569 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:09:16.588 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:09:16.588 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-23 18:08:21.036346549 +0000 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-23 18:02:44.993980088 +0000 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-23 18:02:44.993980088 +0000 2024-04-23T18:09:16.648 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-23 18:02:44.993980088 +0000 2024-04-23T18:09:16.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:09:16.716 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-23T18:09:16.716 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-23T18:09:16.716 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000115715 s, 4.4 MB/s 2024-04-23T18:09:16.718 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:09:16.777 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-04-23T18:09:16.794 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:16.794 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: Reconfiguring daemon mon.c on smithi131 2024-04-23T18:09:16.795 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-23 18:08:21.068346108 +0000 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-23 18:02:45.335975264 +0000 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-23 18:02:45.335975264 +0000 2024-04-23T18:09:16.836 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-23 18:02:45.335975264 +0000 2024-04-23T18:09:16.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:09:16.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:16.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:16.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: Reconfiguring daemon mon.c on smithi131 2024-04-23T18:09:16.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-23T18:09:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-23T18:09:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000128718 s, 4.0 MB/s 2024-04-23T18:09:16.905 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: Reconfiguring daemon mon.c on smithi131 2024-04-23T18:09:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:16.965 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.023 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-23 18:08:21.099345680 +0000 2024-04-23T18:09:17.024 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-23 18:02:45.677970440 +0000 2024-04-23T18:09:17.024 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-23 18:02:45.677970440 +0000 2024-04-23T18:09:17.024 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-23 18:02:45.677970440 +0000 2024-04-23T18:09:17.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:09:17.091 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-23T18:09:17.091 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-23T18:09:17.091 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000143426 s, 3.6 MB/s 2024-04-23T18:09:17.092 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:09:17.151 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-23 18:08:21.130345253 +0000 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-23 18:02:46.020965602 +0000 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-23 18:02:46.020965602 +0000 2024-04-23T18:09:17.209 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-23 18:02:46.020965602 +0000 2024-04-23T18:09:17.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:09:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-23T18:09:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-23T18:09:17.276 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000129235 s, 4.0 MB/s 2024-04-23T18:09:17.278 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:09:17.337 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-23T18:09:17.337 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:09:17.356 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:09:17.356 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-23 18:09:03.560139768 +0000 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-23 18:03:27.890599833 +0000 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-23 18:03:27.890599833 +0000 2024-04-23T18:09:17.414 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-23 18:03:27.890599833 +0000 2024-04-23T18:09:17.415 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:09:17.478 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-23T18:09:17.479 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-23T18:09:17.479 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000105492 s, 4.9 MB/s 2024-04-23T18:09:17.480 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:09:17.538 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.594 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-23 18:09:03.560139768 +0000 2024-04-23T18:09:17.595 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-23 18:03:28.221594574 +0000 2024-04-23T18:09:17.595 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-23 18:03:28.221594574 +0000 2024-04-23T18:09:17.595 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-23 18:03:28.221594574 +0000 2024-04-23T18:09:17.595 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:09:17.658 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-23T18:09:17.659 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-23T18:09:17.659 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000101161 s, 5.1 MB/s 2024-04-23T18:09:17.660 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:09:17.717 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-04-23T18:09:17.773 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:09:17.773 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-23 18:09:03.561139753 +0000 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-23 18:03:28.554589284 +0000 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-23 18:03:28.554589284 +0000 2024-04-23T18:09:17.774 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-23 18:03:28.554589284 +0000 2024-04-23T18:09:17.774 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:09:17.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:17 smithi110 ceph-mon[32585]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi026=x;smithi110=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:17.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:17 smithi110 ceph-mon[32585]: Saving service mgr spec with placement smithi026=x;smithi110=y;smithi131=z;count:3 2024-04-23T18:09:17.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:17 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:17 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:17 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-23T18:09:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-23T18:09:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.8112e-05 s, 5.8 MB/s 2024-04-23T18:09:17.838 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:09:17.894 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-04-23T18:09:17.949 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-23 18:09:03.561139753 +0000 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-23 18:03:28.886584010 +0000 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-23 18:03:28.886584010 +0000 2024-04-23T18:09:17.950 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-23 18:03:28.886584010 +0000 2024-04-23T18:09:17.950 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:09:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:17 smithi026 ceph-mon[29364]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi026=x;smithi110=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:17 smithi026 ceph-mon[29364]: Saving service mgr spec with placement smithi026=x;smithi110=y;smithi131=z;count:3 2024-04-23T18:09:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:17 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:17 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:17 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:17 smithi131 ceph-mon[33041]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi026=x;smithi110=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:17 smithi131 ceph-mon[33041]: Saving service mgr spec with placement smithi026=x;smithi110=y;smithi131=z;count:3 2024-04-23T18:09:17.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:17 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:17 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:17.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:17 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-23T18:09:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-23T18:09:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000108666 s, 4.7 MB/s 2024-04-23T18:09:18.013 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:09:18.069 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-04-23T18:09:18.069 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:09:18.084 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:09:18.085 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-04-23T18:09:18.140 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:09:18.140 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-04-23 18:08:54.076203559 +0000 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-04-23 18:03:23.931773853 +0000 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-04-23 18:03:23.931773853 +0000 2024-04-23T18:09:18.141 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-04-23 18:03:23.931773853 +0000 2024-04-23T18:09:18.141 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:09:18.205 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-04-23T18:09:18.205 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-04-23T18:09:18.205 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000263011 s, 1.9 MB/s 2024-04-23T18:09:18.207 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:09:18.262 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-04-23T18:09:18.317 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:09:18.317 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:18.317 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-23T18:09:18.317 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:18.317 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:18.318 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-04-23 18:08:54.076203559 +0000 2024-04-23T18:09:18.318 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-04-23 18:03:24.280766234 +0000 2024-04-23T18:09:18.318 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-04-23 18:03:24.280766234 +0000 2024-04-23T18:09:18.318 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-04-23 18:03:24.280766234 +0000 2024-04-23T18:09:18.318 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:09:18.379 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-04-23T18:09:18.379 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-04-23T18:09:18.380 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000215782 s, 2.4 MB/s 2024-04-23T18:09:18.381 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:09:18.436 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-04-23 18:08:54.076203559 +0000 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-04-23 18:03:24.622758769 +0000 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-04-23 18:03:24.622758769 +0000 2024-04-23T18:09:18.492 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-04-23 18:03:24.622758769 +0000 2024-04-23T18:09:18.493 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:09:18.555 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-04-23T18:09:18.555 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-04-23T18:09:18.555 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.00022725 s, 2.3 MB/s 2024-04-23T18:09:18.556 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:09:18.612 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-04-23 18:08:54.076203559 +0000 2024-04-23T18:09:18.669 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-04-23 18:03:24.960751391 +0000 2024-04-23T18:09:18.670 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-04-23 18:03:24.960751391 +0000 2024-04-23T18:09:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-04-23 18:03:24.960751391 +0000 2024-04-23T18:09:18.670 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:09:18.732 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-04-23T18:09:18.732 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-04-23T18:09:18.732 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000186563 s, 2.7 MB/s 2024-04-23T18:09:18.733 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:09:18.789 INFO:tasks.cephadm:Deploying osd.0 on smithi026 with /dev/vg_nvme/lv_4... 2024-04-23T18:09:18.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:09:18.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:18.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:18.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:18.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:18.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:18.821 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:18.821 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:18.821 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:18 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:18.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:18 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:18 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:19.041 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:19.829 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:19 smithi026 ceph-mon[29364]: Deploying daemon mgr.z on smithi131 2024-04-23T18:09:19.849 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:19 smithi131 ceph-mon[33041]: Deploying daemon mgr.z on smithi131 2024-04-23T18:09:19.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:19 smithi110 ceph-mon[32585]: Deploying daemon mgr.z on smithi131 2024-04-23T18:09:20.243 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 systemd[1]: Starting Ceph mgr.z for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:09:20.556 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 podman[34899]: 2024-04-23 18:09:20.174537586 +0000 UTC m=+0.017381004 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:09:20.556 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 podman[34899]: 2024-04-23 18:09:20.302571891 +0000 UTC m=+0.145415291 container create 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T18:09:20.556 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 podman[34899]: 2024-04-23 18:09:20.562016783 +0000 UTC m=+0.404860187 container init 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:09:20.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:20 smithi026 ceph-mon[29364]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:20.899 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:20 smithi131 ceph-mon[33041]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 podman[34899]: 2024-04-23 18:09:20.565210867 +0000 UTC m=+0.408054269 container start 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 ceph-mgr[34919]: -- 172.21.15.131:0/1504804927 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561b9d5685a0 con 0x561b9d53d000 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 bash[34899]: 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 systemd[1]: Started Ceph mgr.z for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:20.720+0000 7f490151e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:09:20.900 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:20.905+0000 7f490151e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:20.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:20 smithi110 ceph-mon[32585]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00848415 s, 1.2 GB/s 2024-04-23T18:09:20.975 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-23T18:09:21.242 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:21.007+0000 7f490151e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:09:21.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_4 2024-04-23T18:09:21.706 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:21.320+0000 7f490151e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:09:21.706 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:21.406+0000 7f490151e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:09:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:21.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:21.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:21.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:21.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:21.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: Deploying daemon mgr.y on smithi110 2024-04-23T18:09:21.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:21 smithi110 ceph-mon[32585]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: Deploying daemon mgr.y on smithi110 2024-04-23T18:09:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:21 smithi026 ceph-mon[29364]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:21.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: Deploying daemon mgr.y on smithi110 2024-04-23T18:09:21.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:21 smithi131 ceph-mon[33041]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:22.932 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:22 smithi110 systemd[1]: Starting Ceph mgr.y for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:09:22.977 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:22.631+0000 7f490151e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:09:22.978 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:22.806+0000 7f490151e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:22.978 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:22.983+0000 7f490151e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:09:23.187 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:22 smithi110 podman[33945]: 2024-04-23 18:09:22.938442073 +0000 UTC m=+0.108890611 container create dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T18:09:23.187 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:22 smithi110 podman[33945]: 2024-04-23 18:09:22.860635889 +0000 UTC m=+0.031084440 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:09:23.187 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 podman[33945]: 2024-04-23 18:09:23.165756049 +0000 UTC m=+0.336204601 container init dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:09:23.188 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 podman[33945]: 2024-04-23 18:09:23.168532247 +0000 UTC m=+0.338980788 container start dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-23T18:09:23.188 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 ceph-mgr[33965]: -- 172.21.15.110:0/3705574344 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ef66285a0 con 0x563ef65fc800 2024-04-23T18:09:23.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 bash[33945]: dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 2024-04-23T18:09:23.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 systemd[1]: Started Ceph mgr.y for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:09:23.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:23.329+0000 7fdd712df200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:09:23.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 -- 172.21.15.26:0/811734341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9020102bc0 msgr2=0x7f9020103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 --2- 172.21.15.26:0/811734341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9020102bc0 0x7f9020103020 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90080099c0 tx=0x7f9008009cd0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 -- 172.21.15.26:0/811734341 shutdown_connections 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 --2- 172.21.15.26:0/811734341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020103560 0x7f9020109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 --2- 172.21.15.26:0/811734341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f90201019c0 0x7f9020101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 --2- 172.21.15.26:0/811734341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9020102bc0 0x7f9020103020 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 -- 172.21.15.26:0/811734341 >> 172.21.15.26:0/811734341 conn(0x7f90200fcf60 msgr2=0x7f90200ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 -- 172.21.15.26:0/811734341 shutdown_connections 2024-04-23T18:09:23.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.470+0000 7f9024bb0700 1 -- 172.21.15.26:0/811734341 wait complete. 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 Processor -- start 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 -- start start 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020102bc0 0x7f9020199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9020103560 0x7f90201a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f902010d480 con 0x7f90201019c0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f902010d300 con 0x7f9020102bc0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f9024bb0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f902010d600 con 0x7f9020103560 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f901e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f901e59c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44560/0 (socket says 172.21.15.26:44560) 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f901e59c700 1 -- 172.21.15.26:0/3772843866 learned_addr learned my addr 172.21.15.26:0/3772843866 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f901dd9b700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020102bc0 0x7f9020199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.471+0000 7f901ed9d700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9020103560 0x7f90201a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 -- 172.21.15.26:0/3772843866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9020103560 msgr2=0x7f90201a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9020103560 0x7f90201a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 -- 172.21.15.26:0/3772843866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020102bc0 msgr2=0x7f9020199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020102bc0 0x7f9020199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 -- 172.21.15.26:0/3772843866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90201a0820 con 0x7f90201019c0 2024-04-23T18:09:23.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f901e59c700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9018007f10 tx=0x7f901800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901801a070 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f901800dd20 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90180145e0 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90201a0b10 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.472+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90201a1050 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.473+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f901801d070 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.473+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9020069730 con 0x7f90201019c0 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.473+0000 7f90177fe700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 0x7f900c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.473+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f901804c430 con 0x7f90201019c0 2024-04-23T18:09:23.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.473+0000 7f901dd9b700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 0x7f900c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.474+0000 7f901dd9b700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 0x7f900c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9008000c00 tx=0x7f9008005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.479+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9018010070 con 0x7f90201019c0 2024-04-23T18:09:23.576 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:23.279+0000 7f490151e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:09:23.576 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:23.375+0000 7f490151e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:09:23.576 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:23.581+0000 7f490151e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:09:23.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:23.724+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f90200ff110 con 0x7f900c039890 2024-04-23T18:09:23.942 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:23.522+0000 7fdd712df200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:23.943 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:23.629+0000 7fdd712df200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:09:23.992 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:23.784+0000 7f490151e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:24.278 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:23.954+0000 7fdd712df200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:09:24.279 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:24.045+0000 7fdd712df200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:09:24.572 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:24.573 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:24.574 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:24.425+0000 7f490151e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:09:24.574 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:24.575 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:24.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:24.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.164 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:24.865+0000 7f490151e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:09:25.164 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:24.962+0000 7f490151e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:09:25.164 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:25.169+0000 7f490151e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:09:25.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:25.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.558 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.607 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:25.279+0000 7fdd712df200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:09:25.607 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:25.446+0000 7fdd712df200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:25.607 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:25.612+0000 7fdd712df200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:09:25.607 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:09:25.608 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:25.992 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:25.695+0000 7f490151e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:09:25.993 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:25.787+0000 7f490151e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:09:26.174 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:25.893+0000 7fdd712df200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:09:26.175 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:25.981+0000 7fdd712df200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:09:26.175 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:26.179+0000 7fdd712df200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:09:26.314 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:26.031+0000 7f490151e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:09:26.314 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:26.161+0000 7f490151e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:09:26.314 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:26.256+0000 7f490151e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:09:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-mon[32585]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-mon[32585]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:09:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-mon[32585]: Reconfiguring daemon mgr.x on smithi026 2024-04-23T18:09:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:26.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:26.369+0000 7fdd712df200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:26.672 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-mon[33041]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:26.672 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-mon[33041]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:09:26.672 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-mon[33041]: Reconfiguring daemon mgr.x on smithi026 2024-04-23T18:09:26.672 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:26.672 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:26.371+0000 7f490151e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:26.673 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:26.677+0000 7f490151e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:09:26.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:26 smithi026 ceph-mon[29364]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:26.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:26 smithi026 ceph-mon[29364]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:09:26.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:26 smithi026 ceph-mon[29364]: Reconfiguring daemon mgr.x on smithi026 2024-04-23T18:09:26.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:26 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:27.315 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:26.955+0000 7fdd712df200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:09:27.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]: dispatch 2024-04-23T18:09:27.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]': finished 2024-04-23T18:09:27.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-mon[32585]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:09:27.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:27.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1708419428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:27.652 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:27.366+0000 7fdd712df200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:09:27.652 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:27.460+0000 7fdd712df200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:09:27.652 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:27 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:27.656+0000 7fdd712df200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:09:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]: dispatch 2024-04-23T18:09:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]': finished 2024-04-23T18:09:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:27 smithi026 ceph-mon[29364]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:09:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1708419428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:27.742 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:27.276+0000 7f490151e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:09:27.495+0000 7f490151e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]: dispatch 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2423620768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6232862c-582d-4b53-96c1-a331e0e5849d"}]': finished 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-mon[33041]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:27.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1708419428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:28.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:28.415+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f9018014820 con 0x7f90201019c0 2024-04-23T18:09:28.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: Standby manager daemon z started 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.165+0000 7fdd712df200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:09:28.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.256+0000 7fdd712df200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:09:28.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:28.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: Standby manager daemon z started 2024-04-23T18:09:28.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:09:28.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:28.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:09:28.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:28 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:28.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:28.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: Standby manager daemon z started 2024-04-23T18:09:28.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:09:28.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:28.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:09:28.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:28 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/419398857' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:28.839 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.501+0000 7fdd712df200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:09:28.839 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.633+0000 7fdd712df200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:09:28.840 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.728+0000 7fdd712df200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:09:28.840 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:28 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:28.844+0000 7fdd712df200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:29 smithi110 ceph-mon[32585]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:09:29.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:29 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:09:29.443 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:29 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:29.170+0000 7fdd712df200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:09:29.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:29 smithi026 ceph-mon[29364]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:09:29.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:29 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:09:29.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:29 smithi131 ceph-mon[33041]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:09:29.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:29 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:09:30.192 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:29 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:29.770+0000 7fdd712df200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:09:30.193 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:09:29 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:09:29.981+0000 7fdd712df200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:09:30.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:30.425+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f9018015940 con 0x7f90201019c0 2024-04-23T18:09:30.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:30.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: Standby manager daemon y started 2024-04-23T18:09:30.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:09:30.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:30.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:09:30.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:30 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:30.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:30.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: Standby manager daemon y started 2024-04-23T18:09:30.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:09:30.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:30.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:09:30.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:30 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:30.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:30.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: Standby manager daemon y started 2024-04-23T18:09:30.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:09:30.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:09:30.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:09:30.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:30 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1759969997' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:09:31.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:31 smithi110 ceph-mon[32585]: mgrmap e15: x(active, since 79s), standbys: y, z 2024-04-23T18:09:31.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:09:31.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:31 smithi026 ceph-mon[29364]: mgrmap e15: x(active, since 79s), standbys: y, z 2024-04-23T18:09:31.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:09:31.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:31 smithi131 ceph-mon[33041]: mgrmap e15: x(active, since 79s), standbys: y, z 2024-04-23T18:09:31.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:32.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:32.744 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:34.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:34 smithi110 ceph-mon[32585]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:34.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:09:34.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:34.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:34 smithi026 ceph-mon[29364]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:34.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:09:34.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:34.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:34 smithi131 ceph-mon[33041]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:34.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:09:34.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:35.585 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:35 smithi026 ceph-mon[29364]: Deploying daemon osd.0 on smithi026 2024-04-23T18:09:35.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:35 smithi110 ceph-mon[32585]: Deploying daemon osd.0 on smithi026 2024-04-23T18:09:35.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:35 smithi131 ceph-mon[33041]: Deploying daemon osd.0 on smithi026 2024-04-23T18:09:36.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:36 smithi110 ceph-mon[32585]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:36.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:36 smithi026 ceph-mon[29364]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:36.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:36 smithi131 ceph-mon[33041]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:38.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:38 smithi110 ceph-mon[32585]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:38.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:38 smithi026 ceph-mon[29364]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:38.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:38 smithi131 ceph-mon[33041]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:40.565 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:40 smithi026 ceph-mon[29364]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:40.566 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.566 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.566 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:40.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:40 smithi131 ceph-mon[33041]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:40.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:40.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:40 smithi110 ceph-mon[32585]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:40.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:40.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:42.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:42 smithi026 ceph-mon[29364]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:42 smithi131 ceph-mon[33041]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:42 smithi110 ceph-mon[32585]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:43.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.401+0000 7f90177fe700 1 -- 172.21.15.26:0/3772843866 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f90200ff110 con 0x7f900c039890 2024-04-23T18:09:43.396 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 0 on host 'smithi026' 2024-04-23T18:09:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 msgr2=0x7f900c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 0x7f900c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9008000c00 tx=0x7f9008005fb0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 msgr2=0x7f9020198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:43.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9018007f10 tx=0x7f901800d3b0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 shutdown_connections 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f900c039890 0x7f900c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9020103560 0x7f90201a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f90201019c0 0x7f9020198bc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 --2- 172.21.15.26:0/3772843866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9020102bc0 0x7f9020199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 >> 172.21.15.26:0/3772843866 conn(0x7f90200fcf60 msgr2=0x7f90200fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 shutdown_connections 2024-04-23T18:09:43.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:09:43.403+0000 7f9024bb0700 1 -- 172.21.15.26:0/3772843866 wait complete. 2024-04-23T18:09:43.983 DEBUG:teuthology.orchestra.run.smithi026:osd.0> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T18:09:43.986 INFO:tasks.cephadm:Deploying osd.1 on smithi110 with /dev/vg_nvme/lv_4... 2024-04-23T18:09:43.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:09:44.227 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:09:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:09:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.652 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:09:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:44.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:09:44.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:44.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:45.606 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:45 smithi110 ceph-mon[32585]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:09:45.606 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:45 smithi110 ceph-mon[32585]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:09:45.606 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:45 smithi110 ceph-mon[32585]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:09:45.606 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:45 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:45.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:45 smithi026 ceph-mon[29364]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:09:45.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:45 smithi026 ceph-mon[29364]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:09:45.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:45 smithi026 ceph-mon[29364]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:09:45.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:45 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:45 smithi131 ceph-mon[33041]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:09:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:45 smithi131 ceph-mon[33041]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:09:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:45 smithi131 ceph-mon[33041]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:09:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:45 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.158 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:09:46.158 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:09:46.158 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-23T18:09:46.159 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-23T18:09:46.159 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838195 s, 1.3 GB/s 2024-04-23T18:09:46.159 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-23T18:09:46.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0[41609]: 2024-04-23T18:09:46.198+0000 7f47d712c700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T18:09:46.440 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0[41609]: 2024-04-23T18:09:46.202+0000 7f47cef17700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:09:46.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:46.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:09:46.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:09:46.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:46 smithi026 ceph-mon[29364]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' 2024-04-23T18:09:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:09:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:09:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:46 smithi110 ceph-mon[32585]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' 2024-04-23T18:09:46.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:46.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:09:46.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:09:46.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:46.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:46 smithi131 ceph-mon[33041]: from='osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197]' entity='osd.0' 2024-04-23T18:09:46.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_4 2024-04-23T18:09:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] boot 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:09:47.662 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:47.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:09:47.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:09:47.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:47.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] boot 2024-04-23T18:09:47.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:09:47.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:47.834 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:09:47.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:09:47.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:47.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: osd.0 [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] boot 2024-04-23T18:09:47.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:09:47.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:09:48.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.399+0000 7feaf08aa700 1 -- 172.21.15.110:0/4070593977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec103ca0 msgr2=0x7feaec104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.399+0000 7feaf08aa700 1 --2- 172.21.15.110:0/4070593977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec103ca0 0x7feaec104100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feae00099c0 tx=0x7feae0009cd0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 -- 172.21.15.110:0/4070593977 shutdown_connections 2024-04-23T18:09:48.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 --2- 172.21.15.110:0/4070593977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec104640 0x7feaec10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 --2- 172.21.15.110:0/4070593977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 0x7feaec102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 --2- 172.21.15.110:0/4070593977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec103ca0 0x7feaec104100 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 -- 172.21.15.110:0/4070593977 >> 172.21.15.110:0/4070593977 conn(0x7feaec0fe040 msgr2=0x7feaec100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 -- 172.21.15.110:0/4070593977 shutdown_connections 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.400+0000 7feaf08aa700 1 -- 172.21.15.110:0/4070593977 wait complete. 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 Processor -- start 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 -- start start 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 0x7feaec198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec103ca0 0x7feaec199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7feaec10e540 con 0x7feaec102aa0 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feaec10e3c0 con 0x7feaec104640 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaf08aa700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7feaec10e6c0 con 0x7feaec103ca0 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaead9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaead9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:57666/0 (socket says 172.21.15.110:57666) 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaead9d700 1 -- 172.21.15.110:0/1060481944 learned_addr learned my addr 172.21.15.110:0/1060481944 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-23T18:09:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feae9d9b700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec103ca0 0x7feaec199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.401+0000 7feaea59c700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 0x7feaec198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 -- 172.21.15.110:0/1060481944 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec103ca0 msgr2=0x7feaec199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec103ca0 0x7feaec199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 -- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 msgr2=0x7feaec198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 0x7feaec198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 -- 172.21.15.110:0/1060481944 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec1a0840 con 0x7feaec104640 2024-04-23T18:09:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaead9d700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feae4007ee0 tx=0x7feae400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4004bb0 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae4004d50 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaec1a0b30 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4005950 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.402+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaec1a1070 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.403+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaec069730 con 0x7feaec104640 2024-04-23T18:09:48.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.404+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7feae401a020 con 0x7feaec104640 2024-04-23T18:09:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.404+0000 7feadb7fe700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 0x7fead40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.405+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7feae40cdf40 con 0x7feaec104640 2024-04-23T18:09:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.405+0000 7feaea59c700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 0x7fead40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.405+0000 7feaea59c700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 0x7fead40a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feaec103b00 tx=0x7feadc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.409+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feae4010070 con 0x7feaec104640 2024-04-23T18:09:48.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:09:48.617+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7feaec062e80 con 0x7fead40a3c30 2024-04-23T18:09:48.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi026 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 17093M 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:49.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:49.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi026 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 17093M 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:49.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:49.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi026 2024-04-23T18:09:49.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 17093M 2024-04-23T18:09:49.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:49.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:09:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:50 smithi110 ceph-mon[32585]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:50.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:50 smithi026 ceph-mon[29364]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:50 smithi131 ceph-mon[33041]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:51.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:51 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/1847692048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:51 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:51 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]': finished 2024-04-23T18:09:51.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:51 smithi110 ceph-mon[32585]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:09:51.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:09:51.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:51 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/1847692048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:51 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:51 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]': finished 2024-04-23T18:09:51.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:51 smithi026 ceph-mon[29364]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:09:51.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:09:51.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:51 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/1847692048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:51 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]: dispatch 2024-04-23T18:09:51.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:51 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea35a550-d935-4416-9e66-29ae820b9e4a"}]': finished 2024-04-23T18:09:51.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:51 smithi131 ceph-mon[33041]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:09:51.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:09:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:52 smithi110 ceph-mon[32585]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/1616549778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:52.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:52 smithi026 ceph-mon[29364]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:52.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/1616549778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:52 smithi131 ceph-mon[33041]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/1616549778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:09:54.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:54 smithi110 ceph-mon[32585]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:54.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:54 smithi026 ceph-mon[29364]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:54.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:54 smithi131 ceph-mon[33041]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:56.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:56 smithi110 ceph-mon[32585]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:56 smithi026 ceph-mon[29364]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:56.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:56 smithi131 ceph-mon[33041]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:58.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:58 smithi110 ceph-mon[32585]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:58.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:09:58.925 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:58 smithi026 ceph-mon[29364]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:09:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:58 smithi131 ceph-mon[33041]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:09:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:09:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:59.927 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:09:59 smithi110 ceph-mon[32585]: Deploying daemon osd.1 on smithi110 2024-04-23T18:09:59.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:09:59 smithi026 ceph-mon[29364]: Deploying daemon osd.1 on smithi110 2024-04-23T18:09:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:09:59 smithi131 ceph-mon[33041]: Deploying daemon osd.1 on smithi110 2024-04-23T18:10:00.928 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:00 smithi110 ceph-mon[32585]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:00.928 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:00 smithi110 ceph-mon[32585]: overall HEALTH_OK 2024-04-23T18:10:00.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:00 smithi026 ceph-mon[29364]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:00.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:00 smithi026 ceph-mon[29364]: overall HEALTH_OK 2024-04-23T18:10:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:00 smithi131 ceph-mon[33041]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:00 smithi131 ceph-mon[33041]: overall HEALTH_OK 2024-04-23T18:10:02.773 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:02 smithi110 ceph-mon[32585]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:02.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:02 smithi026 ceph-mon[29364]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:02.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:02 smithi131 ceph-mon[33041]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:04 smithi110 ceph-mon[32585]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:04 smithi026 ceph-mon[29364]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:04 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:04 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:04 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:04 smithi131 ceph-mon[33041]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:04 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:04 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:04 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:06.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:06.946 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.946 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.946 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:06.946 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:06.946 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:06.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:06.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:06.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:06.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:06.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.243+0000 7feadb7fe700 1 -- 172.21.15.110:0/1060481944 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7feaec062e80 con 0x7fead40a3c30 2024-04-23T18:10:07.237 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 1 on host 'smithi110' 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 msgr2=0x7fead40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 0x7fead40a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feaec103b00 tx=0x7feadc00b410 comp rx=0 tx=0).stop 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 msgr2=0x7feaec1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feae4007ee0 tx=0x7feae400d3b0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 shutdown_connections 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fead40a3c30 0x7fead40a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feaec104640 0x7feaec1a01a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7feaec102aa0 0x7feaec198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 --2- 172.21.15.110:0/1060481944 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feaec103ca0 0x7feaec199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 >> 172.21.15.110:0/1060481944 conn(0x7feaec0fe040 msgr2=0x7feaec107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 shutdown_connections 2024-04-23T18:10:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:10:07.245+0000 7feaf08aa700 1 -- 172.21.15.110:0/1060481944 wait complete. 2024-04-23T18:10:07.799 DEBUG:teuthology.orchestra.run.smithi110:osd.1> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T18:10:07.802 INFO:tasks.cephadm:Deploying osd.2 on smithi131 with /dev/vg_nvme/lv_4... 2024-04-23T18:10:07.802 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:10:08.047 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:08.442 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:08 smithi131 ceph-mon[33041]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:08.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:08 smithi110 ceph-mon[32585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:08.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:08.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:08.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:08.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:08.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:08 smithi026 ceph-mon[29364]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:10:09.272 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:09 smithi131 ceph-mon[33041]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:10:09.272 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:09 smithi131 ceph-mon[33041]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.272 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:09 smithi131 ceph-mon[33041]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:10:09.273 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:09.273 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:09 smithi131 ceph-mon[33041]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-mon[32585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:10:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-mon[32585]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-mon[32585]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:10:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-mon[32585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.443 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1[37156]: 2024-04-23T18:10:09.241+0000 7faf91f88700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T18:10:09.443 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:09 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1[37156]: 2024-04-23T18:10:09.246+0000 7faf8dd7b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:09.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:09 smithi026 ceph-mon[29364]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:10:09.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:09 smithi026 ceph-mon[29364]: from='osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:09 smithi026 ceph-mon[29364]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:10:09.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:09.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:09 smithi026 ceph-mon[29364]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:10:09.823 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:10:09.824 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:10:09.824 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-04-23T18:10:09.824 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-04-23T18:10:09.824 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112473 s, 932 MB/s 2024-04-23T18:10:09.824 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-04-23T18:10:10.418 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:10 smithi131 ceph-mon[33041]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:10.418 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:10 smithi131 ceph-mon[33041]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:10:10.418 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:10 smithi131 ceph-mon[33041]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:10:10.418 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:10.418 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:10.419 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi131:vg_nvme/lv_4 2024-04-23T18:10:10.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:10 smithi110 ceph-mon[32585]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:10.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:10 smithi110 ceph-mon[32585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:10:10.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:10 smithi110 ceph-mon[32585]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:10:10.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:10 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:10.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:10 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:10.662 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:10.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:10 smithi026 ceph-mon[29364]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:10:10.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:10 smithi026 ceph-mon[29364]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:10:10.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:10 smithi026 ceph-mon[29364]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:10:10.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:10.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] boot 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi110 2024-04-23T18:10:11.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:11.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] boot 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:11.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi110 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:11.605 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: osd.1 [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] boot 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi110 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-23T18:10:11.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:11.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:11.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:11.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:12.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/819426803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e84105ea0 msgr2=0x7f1e84108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:12.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/819426803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e84105ea0 0x7f1e84108290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009990 tx=0x7f1e70009ca0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/819426803 shutdown_connections 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/819426803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e841087d0 0x7f1e8410ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/819426803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84103570 0x7f1e84105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/819426803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e84105ea0 0x7f1e84108290 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/819426803 >> 172.21.15.131:0/819426803 conn(0x7f1e840fcfa0 msgr2=0x7f1e840ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/819426803 shutdown_connections 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.194+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/819426803 wait complete. 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 Processor -- start 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 -- start start 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e84103570 0x7f1e84198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84105ea0 0x7f1e84199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:12.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8410e570 con 0x7f1e84103570 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8410e3f0 con 0x7f1e84105ea0 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e8b1f6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8410e6f0 con 0x7f1e841087d0 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e89793700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e89793700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:43324/0 (socket says 172.21.15.131:43324) 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e89793700 1 -- 172.21.15.131:0/4214284375 learned_addr learned my addr 172.21.15.131:0/4214284375 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e83fff700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84105ea0 0x7f1e84199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.195+0000 7f1e88f92700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e84103570 0x7f1e84198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 -- 172.21.15.131:0/4214284375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84105ea0 msgr2=0x7f1e84199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84105ea0 0x7f1e84199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 -- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e84103570 msgr2=0x7f1e84198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:12.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e84103570 0x7f1e84198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 -- 172.21.15.131:0/4214284375 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e841a0830 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e89793700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c007d10 tx=0x7f1e7c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c016020 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e841013c0 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e7c00acb0 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c0115e0 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.196+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e841018b0 con 0x7f1e841087d0 2024-04-23T18:10:12.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.197+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e84069730 con 0x7f1e841087d0 2024-04-23T18:10:12.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.198+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f1e7c019020 con 0x7f1e841087d0 2024-04-23T18:10:12.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.198+0000 7f1e81ffb700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 0x7f1e740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:12.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.199+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f1e7c0cea50 con 0x7f1e841087d0 2024-04-23T18:10:12.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.199+0000 7f1e88f92700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 0x7f1e740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:12.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.199+0000 7f1e88f92700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 0x7f1e740a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e78005950 tx=0x7f1e78010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:12.197 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.203+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e7c020020 con 0x7f1e841087d0 2024-04-23T18:10:12.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:12.408+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1e84062e80 con 0x7f1e740a3bb0 2024-04-23T18:10:12.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:12 smithi026 ceph-mon[29364]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:12.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:12 smithi026 ceph-mon[29364]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:10:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:12 smithi131 ceph-mon[33041]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:12 smithi131 ceph-mon[33041]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:10:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:12 smithi110 ceph-mon[32585]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:12.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:12 smithi110 ceph-mon[32585]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:10:12.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:12.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:12.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:13.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:13 smithi026 ceph-mon[29364]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:13 smithi131 ceph-mon[33041]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:13 smithi110 ceph-mon[32585]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:14.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:14 smithi026 ceph-mon[29364]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:14.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:14 smithi131 ceph-mon[33041]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:14 smithi110 ceph-mon[32585]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:15.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:15 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/792377092' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:15 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:15 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]': finished 2024-04-23T18:10:15.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:15 smithi131 ceph-mon[33041]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:10:15.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:15.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:15 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/792377092' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:15 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:15 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]': finished 2024-04-23T18:10:15.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:15 smithi110 ceph-mon[32585]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:10:15.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:15 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/792377092' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:15 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]: dispatch 2024-04-23T18:10:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:15 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b22982-126d-46bf-9e92-d1792b8c8477"}]': finished 2024-04-23T18:10:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:15 smithi026 ceph-mon[29364]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:10:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:16 smithi131 ceph-mon[33041]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/1131395008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:16 smithi110 ceph-mon[32585]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/1131395008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:16 smithi026 ceph-mon[29364]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/1131395008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:18.716 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:18 smithi131 ceph-mon[33041]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:18 smithi110 ceph-mon[32585]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:18 smithi026 ceph-mon[29364]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:20.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:20 smithi131 ceph-mon[33041]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:20.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:20 smithi110 ceph-mon[32585]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:20.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:20 smithi026 ceph-mon[29364]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:22.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:22 smithi131 ceph-mon[33041]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:22.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:10:22.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:22.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:22 smithi110 ceph-mon[32585]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:22.745 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:10:22.745 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:22 smithi026 ceph-mon[29364]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:10:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:23.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:23 smithi131 ceph-mon[33041]: Deploying daemon osd.2 on smithi131 2024-04-23T18:10:23.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:23 smithi110 ceph-mon[32585]: Deploying daemon osd.2 on smithi131 2024-04-23T18:10:23.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:23 smithi026 ceph-mon[29364]: Deploying daemon osd.2 on smithi131 2024-04-23T18:10:24.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:24 smithi131 ceph-mon[33041]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:24.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:24 smithi110 ceph-mon[32585]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:24 smithi026 ceph-mon[29364]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:26.584 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:26 smithi131 ceph-mon[33041]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:26.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:26 smithi110 ceph-mon[32585]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:26.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:26 smithi026 ceph-mon[29364]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:28.427 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:28 smithi131 ceph-mon[33041]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:28.427 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.428 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.428 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:28.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:28 smithi110 ceph-mon[32585]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:28 smithi026 ceph-mon[29364]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:28.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:30.613 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:30.613 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.613 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.613 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:30.613 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:30.614 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.669 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 2 on host 'smithi131' 2024-04-23T18:10:30.669 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.674+0000 7f1e81ffb700 1 -- 172.21.15.131:0/4214284375 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1e84062e80 con 0x7f1e740a3bb0 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 msgr2=0x7f1e740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 0x7f1e740a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e78005950 tx=0x7f1e78010040 comp rx=0 tx=0).stop 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 msgr2=0x7f1e841a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c007d10 tx=0x7f1e7c00a340 comp rx=0 tx=0).stop 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 shutdown_connections 2024-04-23T18:10:30.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1e740a3bb0 0x7f1e740a6070 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e841087d0 0x7f1e841a0190 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1e84103570 0x7f1e84198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 --2- 172.21.15.131:0/4214284375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e84105ea0 0x7f1e84199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.676+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 >> 172.21.15.131:0/4214284375 conn(0x7f1e840fcfa0 msgr2=0x7f1e84103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.677+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 shutdown_connections 2024-04-23T18:10:30.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:10:30.677+0000 7f1e8b1f6700 1 -- 172.21.15.131:0/4214284375 wait complete. 2024-04-23T18:10:30.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:30.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:30.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:30.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:30.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:30.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:30.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:30.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.284 DEBUG:teuthology.orchestra.run.smithi131:osd.2> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T18:10:31.287 INFO:tasks.cephadm:Deploying osd.3 on smithi026 with /dev/vg_nvme/lv_3... 2024-04-23T18:10:31.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:10:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:31.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:31.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:31 smithi026 ceph-mon[29364]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:31.914 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:31 smithi026 ceph-mon[29364]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:31.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:31 smithi110 ceph-mon[32585]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:31 smithi110 ceph-mon[32585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:31 smithi131 ceph-mon[33041]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:31 smithi131 ceph-mon[33041]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:10:32.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:10:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:10:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:32.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:32 smithi131 ceph-mon[33041]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:32.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:32 smithi026 ceph-mon[29364]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: from='osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:32.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:32 smithi110 ceph-mon[32585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:10:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:10:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:10:33.632 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-23T18:10:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-23T18:10:33.633 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00811894 s, 1.3 GB/s 2024-04-23T18:10:33.633 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-23T18:10:33.694 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-mon[33041]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:10:33.694 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-mon[33041]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:10:33.695 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.695 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.695 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-mon[33041]: from='osd.2 ' entity='osd.2' 2024-04-23T18:10:33.695 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2[38188]: 2024-04-23T18:10:33.569+0000 7f2266cfb700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T18:10:33.695 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2[38188]: 2024-04-23T18:10:33.700+0000 7f225eae6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:33.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:33 smithi110 ceph-mon[32585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:10:33.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:33 smithi110 ceph-mon[32585]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:10:33.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:33 smithi110 ceph-mon[32585]: from='osd.2 ' entity='osd.2' 2024-04-23T18:10:33.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:33 smithi026 ceph-mon[29364]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:10:33.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:33 smithi026 ceph-mon[29364]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:10:33.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:33.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:33 smithi026 ceph-mon[29364]: from='osd.2 ' entity='osd.2' 2024-04-23T18:10:34.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_3 2024-04-23T18:10:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:34.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:10:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:34.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi131 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 17088M 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] boot 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:35.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:35.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi131 2024-04-23T18:10:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 17088M 2024-04-23T18:10:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] boot 2024-04-23T18:10:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:10:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi131 2024-04-23T18:10:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 17088M 2024-04-23T18:10:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: osd.2 [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] boot 2024-04-23T18:10:35.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:10:35.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:10:35.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 -- 172.21.15.26:0/462592375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c1087a0 msgr2=0x7f427c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 --2- 172.21.15.26:0/462592375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c1087a0 0x7f427c10ac10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f427400b0a0 tx=0x7f427400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 -- 172.21.15.26:0/462592375 shutdown_connections 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 --2- 172.21.15.26:0/462592375 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c1087a0 0x7f427c10ac10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 --2- 172.21.15.26:0/462592375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c103540 0x7f427c105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 --2- 172.21.15.26:0/462592375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c105e70 0x7f427c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 -- 172.21.15.26:0/462592375 >> 172.21.15.26:0/462592375 conn(0x7f427c0fcf70 msgr2=0x7f427c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 -- 172.21.15.26:0/462592375 shutdown_connections 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 -- 172.21.15.26:0/462592375 wait complete. 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.169+0000 7f4284613700 1 Processor -- start 2024-04-23T18:10:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 -- start start 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 0x7f427c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c1087a0 0x7f427c1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10e540 con 0x7f427c103540 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10e3c0 con 0x7f427c105e70 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4284613700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10e6c0 con 0x7f427c1087a0 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41824/0 (socket says 172.21.15.26:41824) 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 -- 172.21.15.26:0/2101025261 learned_addr learned my addr 172.21.15.26:0/2101025261 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:10:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4281bae700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 0x7f427c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4282bb0700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c1087a0 0x7f427c1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 -- 172.21.15.26:0/2101025261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c1087a0 msgr2=0x7f427c1a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c1087a0 0x7f427c1a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 -- 172.21.15.26:0/2101025261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 msgr2=0x7f427c199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 0x7f427c199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 -- 172.21.15.26:0/2101025261 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c1a0880 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f4281bae700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 0x7f427c199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.170+0000 7f42823af700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f426c00d980 tx=0x7f426c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.171+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c016070 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.171+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f426c0044d0 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.171+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c1a0b70 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.171+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c004d00 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.171+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c1a10b0 con 0x7f427c103540 2024-04-23T18:10:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.172+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427c069730 con 0x7f427c103540 2024-04-23T18:10:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.174+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f426c004020 con 0x7f427c103540 2024-04-23T18:10:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.174+0000 7f42737fe700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 0x7f42680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.174+0000 7f4281bae700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 0x7f42680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.174+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f426c0cf3e0 con 0x7f427c103540 2024-04-23T18:10:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.175+0000 7f4281bae700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 0x7f42680a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4278005e10 tx=0x7f4278005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.178+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f426c01f020 con 0x7f427c103540 2024-04-23T18:10:36.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:36.396+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f427c062f40 con 0x7f42680a3bb0 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:36.936 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:36.937 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:36.937 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:36.937 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:36.937 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41807]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:36.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:36.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:36.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:10:36.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:36.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:36.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:36.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:36.943 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41008]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:36.943 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41008]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:36.943 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41008]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:36.943 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41008]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:36.943 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41011]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:36.944 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:36.944 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41011]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:36.944 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:10:36 smithi110 sudo[41011]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:36.959 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45931]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:36.959 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45931]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:36.959 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45931]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:36.960 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45931]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:36.960 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45934]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:36.960 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45934]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:36.960 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45934]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:36.960 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45934]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:37.242 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:37.242 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.242 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41810]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.243 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:10:36 smithi131 sudo[41810]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41813]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41813]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41813]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41813]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41816]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41816]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41816]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 sudo[41816]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.277 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45993]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:37.277 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.277 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45993]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.278 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:36 smithi026 sudo[45993]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.278 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 sudo[45996]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:37.278 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 sudo[45996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.278 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 sudo[45996]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.278 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 sudo[45996]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41014]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:10:37.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 sudo[41017]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:10:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:37.700+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f426c0991f0 con 0x7f427c103540 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:37 smithi110 ceph-mon[32585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:37 smithi026 ceph-mon[29364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:37.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:10:37.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:10:37.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:37.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:10:37.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:37.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:37.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:37 smithi131 ceph-mon[33041]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:10:38.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:38 smithi110 ceph-mon[32585]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:38 smithi110 ceph-mon[32585]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:10:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:38 smithi110 ceph-mon[32585]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-23T18:10:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:38 smithi026 ceph-mon[29364]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:38 smithi026 ceph-mon[29364]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:10:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:38 smithi026 ceph-mon[29364]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-23T18:10:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:38 smithi131 ceph-mon[33041]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:38 smithi131 ceph-mon[33041]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:10:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:38 smithi131 ceph-mon[33041]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-23T18:10:39.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:39 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]: dispatch 2024-04-23T18:10:39.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:39 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]': finished 2024-04-23T18:10:39.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:39 smithi110 ceph-mon[32585]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:10:39.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:39 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]: dispatch 2024-04-23T18:10:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:39 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]': finished 2024-04-23T18:10:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:39 smithi026 ceph-mon[29364]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:10:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:39.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:39 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]: dispatch 2024-04-23T18:10:39.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:39 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1589201920' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f222d1e-da51-4d64-8e76-53e764356085"}]': finished 2024-04-23T18:10:39.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:39 smithi131 ceph-mon[33041]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:10:39.995 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:40 smithi026 ceph-mon[29364]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:40 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1608612564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:40.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:40 smithi131 ceph-mon[33041]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:40.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:40 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1608612564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:41.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:40 smithi110 ceph-mon[32585]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:41.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:40 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1608612564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:42.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:42 smithi026 ceph-mon[29364]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:42.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:42 smithi131 ceph-mon[33041]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:43.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:42 smithi110 ceph-mon[32585]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:44.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:44 smithi026 ceph-mon[29364]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:44.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:44 smithi131 ceph-mon[33041]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:45.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:44 smithi110 ceph-mon[32585]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:46 smithi026 ceph-mon[29364]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:10:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:46.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:46 smithi131 ceph-mon[33041]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:46.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:10:46.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:47.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:46 smithi110 ceph-mon[32585]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:10:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:47 smithi026 ceph-mon[29364]: Deploying daemon osd.3 on smithi026 2024-04-23T18:10:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:47 smithi131 ceph-mon[33041]: Deploying daemon osd.3 on smithi026 2024-04-23T18:10:48.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:47 smithi110 ceph-mon[32585]: Deploying daemon osd.3 on smithi026 2024-04-23T18:10:48.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:48 smithi026 ceph-mon[29364]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:48.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:48 smithi131 ceph-mon[33041]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:49.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:48 smithi110 ceph-mon[32585]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:49.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:49 smithi131 ceph-mon[33041]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:50.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:49 smithi026 ceph-mon[29364]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:50.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:49 smithi110 ceph-mon[32585]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:51.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:51 smithi131 ceph-mon[33041]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:52.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:51 smithi110 ceph-mon[32585]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:51 smithi026 ceph-mon[29364]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:53.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:53.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:53.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:53.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:54.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:53 smithi026 ceph-mon[29364]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:54.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:53 smithi131 ceph-mon[33041]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:54.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:53 smithi110 ceph-mon[32585]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:55.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.607+0000 7f42737fe700 1 -- 172.21.15.26:0/2101025261 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f427c062f40 con 0x7f42680a3bb0 2024-04-23T18:10:55.602 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 3 on host 'smithi026' 2024-04-23T18:10:55.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 msgr2=0x7f42680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 0x7f42680a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4278005e10 tx=0x7f4278005d80 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 msgr2=0x7f427c198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f426c00d980 tx=0x7f426c00dd40 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 shutdown_connections 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42680a3bb0 0x7f42680a6070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f427c1087a0 0x7f427c1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f427c103540 0x7f427c198c30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 --2- 172.21.15.26:0/2101025261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f427c105e70 0x7f427c199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.609+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 >> 172.21.15.26:0/2101025261 conn(0x7f427c0fcf70 msgr2=0x7f427c103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.610+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 shutdown_connections 2024-04-23T18:10:55.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:10:55.610+0000 7f4284613700 1 -- 172.21.15.26:0/2101025261 wait complete. 2024-04-23T18:10:55.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:56.330 DEBUG:teuthology.orchestra.run.smithi026:osd.3> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T18:10:56.332 INFO:tasks.cephadm:Deploying osd.4 on smithi110 with /dev/vg_nvme/lv_3... 2024-04-23T18:10:56.332 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:10:56.577 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:10:56.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:56 smithi026 ceph-mon[29364]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:10:56.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:56 smithi131 ceph-mon[33041]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:10:56.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:56 smithi110 ceph-mon[32585]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:10:57.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:57 smithi110 ceph-mon[32585]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:10:57.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:57 smithi110 ceph-mon[32585]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:10:57.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:57 smithi110 ceph-mon[32585]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:10:57.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:57 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:57.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:57 smithi110 ceph-mon[32585]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:57.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-mon[29364]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:10:57.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-mon[29364]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:10:57.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-mon[29364]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:10:57.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:57.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-mon[29364]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:57.944 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:10:57.752+0000 7f57daf7d700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T18:10:57.944 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:10:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:10:57.760+0000 7f57d6d70700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:10:58.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:57 smithi131 ceph-mon[33041]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:10:58.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:57 smithi131 ceph-mon[33041]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:10:58.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:57 smithi131 ceph-mon[33041]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:10:58.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:57 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:58.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:57 smithi131 ceph-mon[33041]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:10:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:10:58.483 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-23T18:10:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-23T18:10:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00855332 s, 1.2 GB/s 2024-04-23T18:10:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-23T18:10:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:58 smithi026 ceph-mon[29364]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:10:58.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:58 smithi026 ceph-mon[29364]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:10:58.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:58.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.088 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:58 smithi110 ceph-mon[32585]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:10:59.089 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:58 smithi110 ceph-mon[32585]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:10:59.089 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.089 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.117 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_3 2024-04-23T18:10:59.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:58 smithi131 ceph-mon[33041]: from='osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:10:59.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:58 smithi131 ceph-mon[33041]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:10:59.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.354 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] boot 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:10:59.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi026 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 8546M 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:59.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:10:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] boot 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:11:00.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi026 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:00.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:00.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 8546M 2024-04-23T18:11:00.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:00.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:00.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:10:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:11:00.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: osd.3 [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] boot 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi026 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 8546M 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:00.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:00.244 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:10:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.836+0000 7f62841ec700 1 -- 172.21.15.110:0/2100661191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c1025e0 msgr2=0x7f627c109e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.836+0000 7f62841ec700 1 --2- 172.21.15.110:0/2100661191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c1025e0 0x7f627c109e80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f627400b0a0 tx=0x7f627400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- 172.21.15.110:0/2100661191 shutdown_connections 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- 172.21.15.110:0/2100661191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c1025e0 0x7f627c109e80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- 172.21.15.110:0/2100661191 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- 172.21.15.110:0/2100661191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c101c20 0x7f627c1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- 172.21.15.110:0/2100661191 >> 172.21.15.110:0/2100661191 conn(0x7f627c0fc6b0 msgr2=0x7f627c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- 172.21.15.110:0/2100661191 shutdown_connections 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- 172.21.15.110:0/2100661191 wait complete. 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 Processor -- start 2024-04-23T18:11:00.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- start start 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c1025e0 0x7f627c19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d530 con 0x7f627c102fc0 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d3b0 con 0x7f627c101c20 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f62841ec700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d6b0 con 0x7f627c1025e0 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f6281f88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f6281f88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34344/0 (socket says 172.21.15.110:34344) 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.837+0000 7f6281f88700 1 -- 172.21.15.110:0/1629768244 learned_addr learned my addr 172.21.15.110:0/1629768244 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6282789700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281787700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c1025e0 0x7f627c19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:00.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 -- 172.21.15.110:0/1629768244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c1025e0 msgr2=0x7f627c19da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c1025e0 0x7f627c19da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 -- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 msgr2=0x7f627c1a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 -- 172.21.15.110:0/1629768244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c1a0930 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6282789700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6281f88700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f627800b500 tx=0x7f627800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278015020 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6278011ce0 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c1a0c20 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278010d60 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.838+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c1a1160 con 0x7f627c101c20 2024-04-23T18:11:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.839+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f627c069730 con 0x7f627c101c20 2024-04-23T18:11:00.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.840+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6278018020 con 0x7f627c101c20 2024-04-23T18:11:00.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.841+0000 7f6272ffd700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 0x7f62680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:00.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.841+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f62780cea50 con 0x7f627c101c20 2024-04-23T18:11:00.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.841+0000 7f6281787700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 0x7f62680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:00.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.841+0000 7f6281787700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 0x7f62680a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f626c006fd0 tx=0x7f626c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:00.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:00.846+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f627801f020 con 0x7f627c101c20 2024-04-23T18:11:01.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:01.049+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f627c062e80 con 0x7f62680a3c40 2024-04-23T18:11:01.043 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:00 smithi110 ceph-mon[32585]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:11:01.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:00 smithi026 ceph-mon[29364]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:11:01.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:00 smithi131 ceph-mon[33041]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:01.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:01 smithi110 ceph-mon[32585]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:11:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:11:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:02.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:02.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:02.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:01 smithi026 ceph-mon[29364]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:11:02.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:02.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:01 smithi131 ceph-mon[33041]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:11:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:03 smithi110 ceph-mon[32585]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:11:03.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:03 smithi026 ceph-mon[29364]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:11:03.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:03 smithi131 ceph-mon[33041]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:11:04.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/966400813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]': finished 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:04.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/2727654421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/966400813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]': finished 2024-04-23T18:11:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:11:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:04.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/2727654421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/966400813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]: dispatch 2024-04-23T18:11:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3834c24f-b7b4-41f8-8878-1c54b691ff48"}]': finished 2024-04-23T18:11:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:11:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:04.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:04 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/2727654421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:06.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:06 smithi110 ceph-mon[32585]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:06 smithi026 ceph-mon[29364]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:06 smithi131 ceph-mon[33041]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:08.898 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:08 smithi110 ceph-mon[32585]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:08.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:08 smithi026 ceph-mon[29364]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:08.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:08 smithi131 ceph-mon[33041]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:11:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:10 smithi110 ceph-mon[32585]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-23T18:11:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:10 smithi026 ceph-mon[29364]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-23T18:11:10.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:10 smithi131 ceph-mon[33041]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-23T18:11:11.780 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:11:11.781 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:11.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:11:11.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:11:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:12.895 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:12 smithi110 ceph-mon[32585]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T18:11:12.895 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:12 smithi110 ceph-mon[32585]: Deploying daemon osd.4 on smithi110 2024-04-23T18:11:12.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:12 smithi026 ceph-mon[29364]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T18:11:12.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:12 smithi026 ceph-mon[29364]: Deploying daemon osd.4 on smithi110 2024-04-23T18:11:12.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:12 smithi131 ceph-mon[33041]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T18:11:12.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:12 smithi131 ceph-mon[33041]: Deploying daemon osd.4 on smithi110 2024-04-23T18:11:14.770 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:14 smithi110 ceph-mon[32585]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:14 smithi026 ceph-mon[29364]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:14 smithi131 ceph-mon[33041]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:11:16.885 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:16 smithi110 ceph-mon[32585]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:11:16.885 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.885 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.885 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:16 smithi026 ceph-mon[29364]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:11:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:16 smithi131 ceph-mon[33041]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T18:11:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:18 smithi110 ceph-mon[32585]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:11:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:18 smithi026 ceph-mon[29364]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:11:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:18 smithi131 ceph-mon[33041]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:11:19.759 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 4 on host 'smithi110' 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.765+0000 7f6272ffd700 1 -- 172.21.15.110:0/1629768244 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f627c062e80 con 0x7f62680a3c40 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 msgr2=0x7f62680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 0x7f62680a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f626c006fd0 tx=0x7f626c008040 comp rx=0 tx=0).stop 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 msgr2=0x7f627c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f627800b500 tx=0x7f627800b8c0 comp rx=0 tx=0).stop 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 shutdown_connections 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f62680a3c40 0x7f62680a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f627c102fc0 0x7f627c1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f627c101c20 0x7f627c101690 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 --2- 172.21.15.110:0/1629768244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f627c1025e0 0x7f627c19da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 >> 172.21.15.110:0/1629768244 conn(0x7f627c0fc6b0 msgr2=0x7f627c105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 shutdown_connections 2024-04-23T18:11:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:11:19.767+0000 7f62841ec700 1 -- 172.21.15.110:0/1629768244 wait complete. 2024-04-23T18:11:20.096 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.097 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:20.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:20.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:20.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:20.424 DEBUG:teuthology.orchestra.run.smithi110:osd.4> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T18:11:20.426 INFO:tasks.cephadm:Deploying osd.5 on smithi131 with /dev/vg_nvme/lv_3... 2024-04-23T18:11:20.426 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:11:20.670 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:21.096 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:20 smithi131 ceph-mon[33041]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:21.097 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:20 smithi131 ceph-mon[33041]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:21.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:20 smithi110 ceph-mon[32585]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:21.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:20 smithi110 ceph-mon[32585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:21.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:20 smithi026 ceph-mon[29364]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:21.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:20 smithi026 ceph-mon[29364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:22.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:21 smithi110 ceph-mon[32585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:21 smithi026 ceph-mon[29364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: from='osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:22.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:21 smithi131 ceph-mon[33041]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011278 s, 930 MB/s 2024-04-23T18:11:22.383 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-04-23T18:11:22.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-mon[32585]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:11:22.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-mon[32585]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:11:22.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:22.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:22.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-mon[32585]: from='osd.4 ' entity='osd.4' 2024-04-23T18:11:22.943 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:11:22.611+0000 7f3380b75700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T18:11:22.943 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:11:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:11:22.618+0000 7f3378960700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:22.968 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi131:vg_nvme/lv_3 2024-04-23T18:11:23.199 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:22 smithi026 ceph-mon[29364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:11:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:22 smithi026 ceph-mon[29364]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:11:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:23.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:22 smithi026 ceph-mon[29364]: from='osd.4 ' entity='osd.4' 2024-04-23T18:11:23.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:22 smithi131 ceph-mon[33041]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:11:23.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:22 smithi131 ceph-mon[33041]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:11:23.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:23.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:23.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:22 smithi131 ceph-mon[33041]: from='osd.4 ' entity='osd.4' 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi110 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-23T18:11:23.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] boot 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:11:23.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi110 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:24.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] boot 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:11:24.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi110 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: osd.4 [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] boot 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:11:24.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 -- 172.21.15.131:0/290653644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8102ba0 msgr2=0x7f3df8103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 --2- 172.21.15.131:0/290653644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8102ba0 0x7f3df8103000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009990 tx=0x7f3dec009ca0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 -- 172.21.15.131:0/290653644 shutdown_connections 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 --2- 172.21.15.131:0/290653644 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8103540 0x7f3df8109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 --2- 172.21.15.131:0/290653644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 0x7f3df8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 --2- 172.21.15.131:0/290653644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8102ba0 0x7f3df8103000 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 -- 172.21.15.131:0/290653644 >> 172.21.15.131:0/290653644 conn(0x7f3df80fcf40 msgr2=0x7f3df80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 -- 172.21.15.131:0/290653644 shutdown_connections 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.718+0000 7f3dffb68700 1 -- 172.21.15.131:0/290653644 wait complete. 2024-04-23T18:11:24.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 Processor -- start 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 -- start start 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 0x7f3df806bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8102ba0 0x7f3df819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810d530 con 0x7f3df8102ba0 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810d3b0 con 0x7f3df81019a0 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dffb68700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810d6b0 con 0x7f3df8103540 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dfe105700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dfe105700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:50422/0 (socket says 172.21.15.131:50422) 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dfd904700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 0x7f3df806bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.719+0000 7f3dfe105700 1 -- 172.21.15.131:0/3743721080 learned_addr learned my addr 172.21.15.131:0/3743721080 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfd103700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8102ba0 0x7f3df819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 -- 172.21.15.131:0/3743721080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 msgr2=0x7f3df806bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 0x7f3df806bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 -- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8102ba0 msgr2=0x7f3df819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8102ba0 0x7f3df819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 -- 172.21.15.131:0/3743721080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df81a0900 con 0x7f3df8103540 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dfe105700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3de800b550 tx=0x7f3de800b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8015020 con 0x7f3df8103540 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df81a0bf0 con 0x7f3df8103540 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3de8011ce0 con 0x7f3df8103540 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8010dd0 con 0x7f3df8103540 2024-04-23T18:11:24.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.720+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df81a10e0 con 0x7f3df8103540 2024-04-23T18:11:24.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.721+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df8069730 con 0x7f3df8103540 2024-04-23T18:11:24.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.722+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3de8018020 con 0x7f3df8103540 2024-04-23T18:11:24.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.723+0000 7f3df6ffd700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 0x7f3de00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:24.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.723+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f3de80cefe0 con 0x7f3df8103540 2024-04-23T18:11:24.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.723+0000 7f3dfd904700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 0x7f3de00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:24.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.723+0000 7f3dfd904700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 0x7f3de00a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3df8102a00 tx=0x7f3df000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:24.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.726+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3de801f020 con 0x7f3df8103540 2024-04-23T18:11:24.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:24.931+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3df8062e80 con 0x7f3de00a3c40 2024-04-23T18:11:26.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:26.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:11:26.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:26.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:26.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:26.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:26.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:11:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:26.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:26.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:26.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:28.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:28.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/2851648742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]': finished 2024-04-23T18:11:28.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:11:28.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:28.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:28.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/2851648742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]': finished 2024-04-23T18:11:28.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:11:28.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/2851648742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]: dispatch 2024-04-23T18:11:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b632a3-1a0b-4918-9a4d-ccc144053d30"}]': finished 2024-04-23T18:11:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:11:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:29.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:28 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/3449315008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:28 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/3449315008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:29.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:28 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/3449315008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:30.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:29 smithi110 ceph-mon[32585]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:30.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:29 smithi026 ceph-mon[29364]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:30.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:29 smithi131 ceph-mon[33041]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:32.082 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:31 smithi131 ceph-mon[33041]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:32.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:31 smithi110 ceph-mon[32585]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:32.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:31 smithi026 ceph-mon[29364]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:34.124 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:33 smithi131 ceph-mon[33041]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:34.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:33 smithi110 ceph-mon[32585]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:34.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:33 smithi026 ceph-mon[29364]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:11:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:35.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:11:35.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:35.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:11:35.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:35 smithi131 ceph-mon[33041]: Deploying daemon osd.5 on smithi131 2024-04-23T18:11:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:35 smithi131 ceph-mon[33041]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:36.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:35 smithi110 ceph-mon[32585]: Deploying daemon osd.5 on smithi131 2024-04-23T18:11:36.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:35 smithi110 ceph-mon[32585]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:36.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:35 smithi026 ceph-mon[29364]: Deploying daemon osd.5 on smithi131 2024-04-23T18:11:36.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:35 smithi026 ceph-mon[29364]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:38.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:37 smithi110 ceph-mon[32585]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:38.201 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:37 smithi131 ceph-mon[33041]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:37 smithi026 ceph-mon[29364]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:40.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:39 smithi110 ceph-mon[32585]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:40.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:39 smithi026 ceph-mon[29364]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:40.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:39 smithi131 ceph-mon[33041]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:41.229 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:41.230 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:41.230 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:41.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:41.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:41.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:41.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:41.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:41.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:42.109 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:42 smithi131 ceph-mon[33041]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:42.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:42 smithi110 ceph-mon[32585]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:42.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:42 smithi026 ceph-mon[29364]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:43.492 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.498+0000 7f3df6ffd700 1 -- 172.21.15.131:0/3743721080 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3df8062e80 con 0x7f3de00a3c40 2024-04-23T18:11:43.493 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 5 on host 'smithi131' 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.500+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 msgr2=0x7f3de00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.500+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 0x7f3de00a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3df8102a00 tx=0x7f3df000d040 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.500+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 msgr2=0x7f3df81a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.500+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3de800b550 tx=0x7f3de800b860 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 shutdown_connections 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3de00a3c40 0x7f3de00a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df8103540 0x7f3df81a0030 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df81019a0 0x7f3df806bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 --2- 172.21.15.131:0/3743721080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df8102ba0 0x7f3df819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:43.495 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 >> 172.21.15.131:0/3743721080 conn(0x7f3df80fcf40 msgr2=0x7f3df8105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:43.496 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 shutdown_connections 2024-04-23T18:11:43.496 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:11:43.501+0000 7f3dffb68700 1 -- 172.21.15.131:0/3743721080 wait complete. 2024-04-23T18:11:43.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:43.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:43.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:43 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:44.053 DEBUG:teuthology.orchestra.run.smithi131:osd.5> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T18:11:44.055 INFO:tasks.cephadm:Deploying osd.6 on smithi026 with /dev/vg_nvme/lv_2... 2024-04-23T18:11:44.056 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:11:44.083 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:43 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:44.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:44.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:43 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:11:44.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:44 smithi131 ceph-mon[33041]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:44.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:44 smithi131 ceph-mon[33041]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:45.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:44 smithi110 ceph-mon[32585]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:45.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:44 smithi110 ceph-mon[32585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:45.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:44 smithi026 ceph-mon[29364]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:45.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:44 smithi026 ceph-mon[29364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:11:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:11:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:11:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:45 smithi026 ceph-mon[29364]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:46.047 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:45 smithi131 ceph-mon[33041]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:46.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:11:46.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:11:46.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: from='osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:46.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:11:46.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:45 smithi110 ceph-mon[32585]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00858347 s, 1.2 GB/s 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 2024-04-23T18:11:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-23T18:11:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:11:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:11:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='osd.5 ' entity='osd.5' 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:46.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:46 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:11:46.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='osd.5 ' entity='osd.5' 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:46.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:46.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:46.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:46.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:46.994 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:11:46.663+0000 7f51bd1d1700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T18:11:46.994 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:11:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:11:46.846+0000 7f51b4fbc700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:47.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_2 2024-04-23T18:11:47.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:11:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:11:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T18:11:47.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='osd.5 ' entity='osd.5' 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:47.194 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:47.195 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:47.195 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:47.195 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:46 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:11:47.340 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:11:48.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:11:48.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:11:48.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi131 2024-04-23T18:11:48.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 8544M 2024-04-23T18:11:48.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:48.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] boot 2024-04-23T18:11:48.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:11:48.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi131 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 8544M 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] boot 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:11:48.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:11:48.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:11:48.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi131 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 8544M 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: osd.5 [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] boot 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:11:48.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:11:48.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 -- 172.21.15.26:0/3086378929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 msgr2=0x7f8318108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 --2- 172.21.15.26:0/3086378929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318108270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f830c009990 tx=0x7f830c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 -- 172.21.15.26:0/3086378929 shutdown_connections 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 --2- 172.21.15.26:0/3086378929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 0x7f831810ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 --2- 172.21.15.26:0/3086378929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 0x7f8318105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 --2- 172.21.15.26:0/3086378929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318108270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 -- 172.21.15.26:0/3086378929 >> 172.21.15.26:0/3086378929 conn(0x7f83180fcf60 msgr2=0x7f83180ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 -- 172.21.15.26:0/3086378929 shutdown_connections 2024-04-23T18:11:48.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.886+0000 7f831d035700 1 -- 172.21.15.26:0/3086378929 wait complete. 2024-04-23T18:11:48.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 Processor -- start 2024-04-23T18:11:48.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 -- start start 2024-04-23T18:11:48.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 0x7f8318198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 0x7f83181a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f831810e550 con 0x7f8318105e80 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.887+0000 7f831d035700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f831810e3d0 con 0x7f83181087b0 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831d035700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f831810e6d0 con 0x7f8318103550 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34064/0 (socket says 172.21.15.26:34064) 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 -- 172.21.15.26:0/3712841933 learned_addr learned my addr 172.21.15.26:0/3712841933 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f8316d9d700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 0x7f8318198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831759e700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 0x7f83181a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:48.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 -- 172.21.15.26:0/3712841933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 msgr2=0x7f8318198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 0x7f8318198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 -- 172.21.15.26:0/3712841933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 msgr2=0x7f83181a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 0x7f83181a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 -- 172.21.15.26:0/3712841933 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83181a0890 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831659c700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f830c0036f0 tx=0x7f830c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c027070 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83181a0b20 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c022470 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83181a0fd0 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.888+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c00f6c0 con 0x7f8318105e80 2024-04-23T18:11:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.889+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f831802df80 con 0x7f8318105e80 2024-04-23T18:11:48.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.891+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f830c02a070 con 0x7f8318105e80 2024-04-23T18:11:48.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.891+0000 7f82fffff700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 0x7f83000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:48.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.891+0000 7f8316d9d700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 0x7f83000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:48.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.892+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f830c0d0810 con 0x7f8318105e80 2024-04-23T18:11:48.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.892+0000 7f8316d9d700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 0x7f83000a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8308005950 tx=0x7f830800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:48.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:48.897+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f830c013070 con 0x7f8318105e80 2024-04-23T18:11:49.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:11:49.115+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8318062e80 con 0x7f83000a3cc0 2024-04-23T18:11:49.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:49 smithi026 ceph-mon[29364]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:11:49.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:49.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:49.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:49.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:49 smithi110 ceph-mon[32585]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:11:49.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:49.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:49.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:49 smithi131 ceph-mon[33041]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:11:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:50.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:50 smithi026 ceph-mon[29364]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:50.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:50 smithi026 ceph-mon[29364]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:50.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:50 smithi026 ceph-mon[29364]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:11:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:50 smithi110 ceph-mon[32585]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:50 smithi110 ceph-mon[32585]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:50 smithi110 ceph-mon[32585]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:11:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:50 smithi131 ceph-mon[33041]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:50 smithi131 ceph-mon[33041]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:50 smithi131 ceph-mon[33041]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:11:51.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:51 smithi110 ceph-mon[32585]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:11:51.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:51 smithi026 ceph-mon[29364]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:11:51.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:51 smithi131 ceph-mon[33041]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:11:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]: dispatch 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]': finished 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:11:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3670748964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:52.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:52.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:11:52.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]: dispatch 2024-04-23T18:11:52.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]': finished 2024-04-23T18:11:52.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:11:52.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:11:52.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3670748964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T18:11:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]: dispatch 2024-04-23T18:11:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2393924378' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5e9d18-f986-4bd6-87b6-ceb84b684bf7"}]': finished 2024-04-23T18:11:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:11:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:11:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3670748964' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:54.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:54 smithi110 ceph-mon[32585]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:54.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:54 smithi026 ceph-mon[29364]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:54.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:54 smithi131 ceph-mon[33041]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:11:56.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:56 smithi110 ceph-mon[32585]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T18:11:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:56 smithi026 ceph-mon[29364]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T18:11:56.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:56 smithi131 ceph-mon[33041]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T18:11:58.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:58 smithi110 ceph-mon[32585]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:58 smithi026 ceph-mon[29364]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:58 smithi131 ceph-mon[33041]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:11:59.936 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:11:59.936 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:11:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:59.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:11:59.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:11:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:11:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:11:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:00.910 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:00 smithi026 ceph-mon[29364]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:00.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:00 smithi026 ceph-mon[29364]: Deploying daemon osd.6 on smithi026 2024-04-23T18:12:00.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:00 smithi110 ceph-mon[32585]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:00.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:00 smithi110 ceph-mon[32585]: Deploying daemon osd.6 on smithi026 2024-04-23T18:12:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:00 smithi131 ceph-mon[33041]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:00 smithi131 ceph-mon[33041]: Deploying daemon osd.6 on smithi026 2024-04-23T18:12:02.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:02 smithi110 ceph-mon[32585]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:02.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:02 smithi026 ceph-mon[29364]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:02.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:02 smithi131 ceph-mon[33041]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:04.837 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:04 smithi026 ceph-mon[29364]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:04.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:04 smithi110 ceph-mon[32585]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:04 smithi131 ceph-mon[33041]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:06 smithi110 ceph-mon[32585]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:06.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:06 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:06.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:06 smithi026 ceph-mon[29364]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:06.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:06 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:06 smithi131 ceph-mon[33041]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:06.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:06 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:08 smithi026 ceph-mon[29364]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:08 smithi110 ceph-mon[32585]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:08 smithi131 ceph-mon[33041]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:09.370 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 6 on host 'smithi026' 2024-04-23T18:12:09.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.376+0000 7f82fffff700 1 -- 172.21.15.26:0/3712841933 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8318062e80 con 0x7f83000a3cc0 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.378+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 msgr2=0x7f83000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.378+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 0x7f83000a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8308005950 tx=0x7f830800a300 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.378+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 msgr2=0x7f8318199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.378+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f830c0036f0 tx=0x7f830c0047b0 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 shutdown_connections 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83000a3cc0 0x7f83000a6180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83181087b0 0x7f83181a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8318103550 0x7f8318198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 --2- 172.21.15.26:0/3712841933 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8318105e80 0x7f8318199180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 >> 172.21.15.26:0/3712841933 conn(0x7f83180fcf60 msgr2=0x7f8318103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 shutdown_connections 2024-04-23T18:12:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:12:09.379+0000 7f831d035700 1 -- 172.21.15.26:0/3712841933 wait complete. 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:09.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:09.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:09.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:09.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:12:09.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:09.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:10.082 DEBUG:teuthology.orchestra.run.smithi026:osd.6> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T18:12:10.085 INFO:tasks.cephadm:Deploying osd.7 on smithi110 with /dev/vg_nvme/lv_2... 2024-04-23T18:12:10.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:12:10.330 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:12:10.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-mon[29364]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:10.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-mon[29364]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:12:10.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-mon[29364]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:12:10.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-mon[29364]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:12:10.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:10.642 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:12:10.459+0000 7f813aab3700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T18:12:10.642 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:12:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:12:10.470+0000 7f81368a6700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:10.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:10 smithi131 ceph-mon[33041]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:10.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:10 smithi131 ceph-mon[33041]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:12:10.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:10 smithi131 ceph-mon[33041]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:12:10.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:10 smithi131 ceph-mon[33041]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:12:10.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:10 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:10.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:10 smithi110 ceph-mon[32585]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:10.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:10 smithi110 ceph-mon[32585]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:12:10.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:10 smithi110 ceph-mon[32585]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:12:10.778 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:10 smithi110 ceph-mon[32585]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:12:10.778 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:10 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:11 smithi026 ceph-mon[29364]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:12:11.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:11 smithi026 ceph-mon[29364]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:12:11.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.717 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:11 smithi110 ceph-mon[32585]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:12:11.717 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:11 smithi110 ceph-mon[32585]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:12:11.717 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.717 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:11 smithi131 ceph-mon[33041]: from='osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:12:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:11 smithi131 ceph-mon[33041]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T18:12:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829657 s, 1.3 GB/s 2024-04-23T18:12:12.231 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-23T18:12:12.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:12:12.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:12:12.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] boot 2024-04-23T18:12:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:12:12.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:12:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:12:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] boot 2024-04-23T18:12:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:12:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:12.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:12:12.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:12:12.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:12:12.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: osd.6 [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] boot 2024-04-23T18:12:12.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:12:12.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:12:12.821 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_2 2024-04-23T18:12:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:12:13.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:12:13.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:13.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:13.695 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:12:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:13.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:13.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:13.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:13.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:13.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:14.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 -- 172.21.15.110:0/75688943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 msgr2=0x7fa09410aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/75688943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa09410aef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00b0d0 tx=0x7fa08c00b3e0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 -- 172.21.15.110:0/75688943 shutdown_connections 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/75688943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa09410aef0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/75688943 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 0x7fa094102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/75688943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 0x7fa094104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 -- 172.21.15.110:0/75688943 >> 172.21.15.110:0/75688943 conn(0x7fa0940fe040 msgr2=0x7fa094100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 -- 172.21.15.110:0/75688943 shutdown_connections 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.538+0000 7fa09b9c0700 1 -- 172.21.15.110:0/75688943 wait complete. 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 Processor -- start 2024-04-23T18:12:14.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 -- start start 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 0x7fa094198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 0x7fa0941990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa09410e540 con 0x7fa094102aa0 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa09410e3c0 con 0x7fa094104640 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09b9c0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa09410e6c0 con 0x7fa094103ca0 2024-04-23T18:12:14.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa099f5d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa099f5d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40224/0 (socket says 172.21.15.110:40224) 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa099f5d700 1 -- 172.21.15.110:0/2972310429 learned_addr learned my addr 172.21.15.110:0/2972310429 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa098f5b700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 0x7fa0941990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.539+0000 7fa09975c700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 0x7fa094198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 -- 172.21.15.110:0/2972310429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 msgr2=0x7fa0941990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 0x7fa0941990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 -- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 msgr2=0x7fa094198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 0x7fa094198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 -- 172.21.15.110:0/2972310429 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0941a07e0 con 0x7fa094104640 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa099f5d700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009040 tx=0x7fa08c004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c01f030 con 0x7fa094104640 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08c003a00 con 0x7fa094104640 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0941a0a70 con 0x7fa094104640 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c0047b0 con 0x7fa094104640 2024-04-23T18:12:14.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.540+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0941a0ec0 con 0x7fa094104640 2024-04-23T18:12:14.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.541+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa094069730 con 0x7fa094104640 2024-04-23T18:12:14.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.542+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa08c003bb0 con 0x7fa094104640 2024-04-23T18:12:14.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.542+0000 7fa08a7fc700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 0x7fa0800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:14.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.543+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fa08c00f020 con 0x7fa094104640 2024-04-23T18:12:14.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.543+0000 7fa09975c700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 0x7fa0800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:14.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.543+0000 7fa09975c700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 0x7fa0800a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa084005fd0 tx=0x7fa084005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:14.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.547+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa08c029020 con 0x7fa094104640 2024-04-23T18:12:14.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:14 smithi110 ceph-mon[32585]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:14.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:14 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi026 2024-04-23T18:12:14.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:14 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 5697M 2024-04-23T18:12:14.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:14 smithi131 ceph-mon[33041]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:14.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:14 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi026 2024-04-23T18:12:14.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:14 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 5697M 2024-04-23T18:12:14.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:14.754+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa094062e80 con 0x7fa0800a3c40 2024-04-23T18:12:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:14 smithi026 ceph-mon[29364]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:14 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi026 2024-04-23T18:12:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:14 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 5697M 2024-04-23T18:12:15.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:15.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:15.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:15.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:15.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:15.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:16.504 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:16 smithi110 ceph-mon[32585]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:16.504 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:16 smithi110 ceph-mon[32585]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:16 smithi131 ceph-mon[33041]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:16 smithi131 ceph-mon[33041]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:16 smithi026 ceph-mon[29364]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:16.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:16 smithi026 ceph-mon[29364]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:17.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/540394850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:17 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:17 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]': finished 2024-04-23T18:12:17.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:17 smithi131 ceph-mon[33041]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:12:17.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:17 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:17.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/540394850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:17 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:17 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]': finished 2024-04-23T18:12:17.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:17 smithi110 ceph-mon[32585]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:12:17.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:17 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/540394850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:17 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]: dispatch 2024-04-23T18:12:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:17 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f090001e-2004-4cf8-bcf2-38a16b9618e6"}]': finished 2024-04-23T18:12:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:17 smithi026 ceph-mon[29364]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:12:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:17 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:18.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:18 smithi131 ceph-mon[33041]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:18.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:18 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/2130779021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:18 smithi110 ceph-mon[32585]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:18 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/2130779021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:18 smithi026 ceph-mon[29364]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:18 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/2130779021' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:20.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:20 smithi131 ceph-mon[33041]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:20.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:20 smithi110 ceph-mon[32585]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:20.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:20 smithi026 ceph-mon[29364]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:22.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:22 smithi131 ceph-mon[33041]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:22.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:22 smithi110 ceph-mon[32585]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:22 smithi026 ceph-mon[29364]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:24.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:24 smithi110 ceph-mon[32585]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:24.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:12:24.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:24.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:24 smithi131 ceph-mon[33041]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:24.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:12:24.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:24 smithi026 ceph-mon[29364]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:12:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:25.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:25 smithi110 ceph-mon[32585]: Deploying daemon osd.7 on smithi110 2024-04-23T18:12:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:25 smithi026 ceph-mon[29364]: Deploying daemon osd.7 on smithi110 2024-04-23T18:12:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:25 smithi131 ceph-mon[33041]: Deploying daemon osd.7 on smithi110 2024-04-23T18:12:26.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:26 smithi110 ceph-mon[32585]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:26.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:26 smithi026 ceph-mon[29364]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:26.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:26 smithi131 ceph-mon[33041]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:28.884 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:28 smithi110 ceph-mon[32585]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:28 smithi026 ceph-mon[29364]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:28 smithi131 ceph-mon[33041]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:30 smithi131 ceph-mon[33041]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:30 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:31.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:30 smithi026 ceph-mon[29364]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:31.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:31.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:31.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:30 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:31.152 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:30 smithi110 ceph-mon[32585]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:31.152 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:31.152 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:31.152 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:30 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:32.031 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:31 smithi026 ceph-mon[29364]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:32.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:31 smithi110 ceph-mon[32585]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:32.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:31 smithi131 ceph-mon[33041]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.563+0000 7fa08a7fc700 1 -- 172.21.15.110:0/2972310429 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa094062e80 con 0x7fa0800a3c40 2024-04-23T18:12:32.559 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 7 on host 'smithi110' 2024-04-23T18:12:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 msgr2=0x7fa0800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 0x7fa0800a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa084005fd0 tx=0x7fa084005dd0 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 msgr2=0x7fa0941a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009040 tx=0x7fa08c004d50 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 shutdown_connections 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa0800a3c40 0x7fa0800a6100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa094104640 0x7fa0941a0140 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa094102aa0 0x7fa094198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 --2- 172.21.15.110:0/2972310429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa094103ca0 0x7fa0941990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 >> 172.21.15.110:0/2972310429 conn(0x7fa0940fe040 msgr2=0x7fa094107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 shutdown_connections 2024-04-23T18:12:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:12:32.566+0000 7fa09b9c0700 1 -- 172.21.15.110:0/2972310429 wait complete. 2024-04-23T18:12:33.394 DEBUG:teuthology.orchestra.run.smithi110:osd.7> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T18:12:33.396 INFO:tasks.cephadm:Deploying osd.8 on smithi131 with /dev/vg_nvme/lv_2... 2024-04-23T18:12:33.396 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:12:33.634 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.635 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.635 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.645 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:12:33.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.942 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:12:33 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:12:33.638+0000 7f086d4b47c0 -1 osd.7 0 log_to_monitors true 2024-04-23T18:12:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:34.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:34 smithi110 ceph-mon[32585]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:34.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:34 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:34.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:34 smithi110 ceph-mon[32585]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:34.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:34 smithi110 ceph-mon[32585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:34.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:34 smithi131 ceph-mon[33041]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:34.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:34 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:34.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:34 smithi131 ceph-mon[33041]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:34.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:34 smithi131 ceph-mon[33041]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:34 smithi026 ceph-mon[29364]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:34 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:34 smithi026 ceph-mon[29364]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:34 smithi026 ceph-mon[29364]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:12:35.709 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:12:35.710 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:12:35.710 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-04-23T18:12:35.710 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-04-23T18:12:35.710 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112371 s, 933 MB/s 2024-04-23T18:12:35.710 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-04-23T18:12:35.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:35 smithi110 ceph-mon[32585]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:12:35.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:35 smithi110 ceph-mon[32585]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:35.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:35 smithi110 ceph-mon[32585]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:12:35.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:35.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:35 smithi110 ceph-mon[32585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:35 smithi026 ceph-mon[29364]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:12:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:35 smithi026 ceph-mon[29364]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:35 smithi026 ceph-mon[29364]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:12:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:35 smithi026 ceph-mon[29364]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:35 smithi131 ceph-mon[33041]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:12:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:35 smithi131 ceph-mon[33041]: from='osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:35 smithi131 ceph-mon[33041]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:12:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:35.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:35 smithi131 ceph-mon[33041]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:12:36.301 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi131:vg_nvme/lv_2 2024-04-23T18:12:36.540 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:12:36.551 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:12:36.310+0000 7f0862b19700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T18:12:36.551 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:12:36.320+0000 7f085a904700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:36.877 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:36.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:12:36.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:12:36.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.878 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:36 smithi110 ceph-mon[32585]: from='osd.7 ' entity='osd.7' 2024-04-23T18:12:36.879 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:36.879 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:12:36.879 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:12:36.879 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.879 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.880 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:36 smithi131 ceph-mon[33041]: from='osd.7 ' entity='osd.7' 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:36 smithi026 ceph-mon[29364]: from='osd.7 ' entity='osd.7' 2024-04-23T18:12:37.764 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] boot 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:12:37.765 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.766 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.767 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] boot 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:12:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: osd.7 [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] boot 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:37.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- 172.21.15.131:0/3044089337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb0103560 msgr2=0x7f3cb0109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:37.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/3044089337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb0103560 0x7f3cb0109e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800b0a0 tx=0x7f3ca800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- 172.21.15.131:0/3044089337 shutdown_connections 2024-04-23T18:12:37.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/3044089337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb0103560 0x7f3cb0109e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/3044089337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb01019c0 0x7f3cb0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/3044089337 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- 172.21.15.131:0/3044089337 >> 172.21.15.131:0/3044089337 conn(0x7f3cb00fcfa0 msgr2=0x7f3cb00ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:37.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- 172.21.15.131:0/3044089337 shutdown_connections 2024-04-23T18:12:37.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- 172.21.15.131:0/3044089337 wait complete. 2024-04-23T18:12:37.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 Processor -- start 2024-04-23T18:12:37.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.975+0000 7f3cb5deb700 1 -- start start 2024-04-23T18:12:37.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:37.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb0103560 0x7f3cb019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb010d4d0 con 0x7f3cb0102bc0 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb010d350 con 0x7f3cb0103560 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3cb5deb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb010d650 con 0x7f3cb01019c0 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:40368/0 (socket says 172.21.15.131:40368) 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 -- 172.21.15.131:0/2115513687 learned_addr learned my addr 172.21.15.131:0/2115513687 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:12:37.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3ca6ffd700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caffff700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb0103560 0x7f3cb019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 -- 172.21.15.131:0/2115513687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb0103560 msgr2=0x7f3cb019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb0103560 0x7f3cb019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 -- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 msgr2=0x7f3cb0198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 -- 172.21.15.131:0/2115513687 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb019f740 con 0x7f3cb01019c0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3ca6ffd700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.976+0000 7f3caf7fe700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000b530 tx=0x7f3ca000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.977+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0015020 con 0x7f3cb01019c0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.977+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ca0011ce0 con 0x7f3cb01019c0 2024-04-23T18:12:37.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.977+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb019fa30 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.977+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0010d60 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.977+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb019ffc0 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.978+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb0069730 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.979+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3ca0018020 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.979+0000 7f3cad7fa700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 0x7f3c9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.980+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f3ca00cf030 con 0x7f3cb01019c0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.980+0000 7f3ca6ffd700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 0x7f3c9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:37.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.980+0000 7f3ca6ffd700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 0x7f3c9c0a6180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c98005950 tx=0x7f3c980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:37.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:37.984+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ca001f020 con 0x7f3cb01019c0 2024-04-23T18:12:38.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:38.189+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3cb0062e80 con 0x7f3c9c0a3cc0 2024-04-23T18:12:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi110 2024-04-23T18:12:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-23T18:12:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:12:38.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:38.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:38.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:38 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi110 2024-04-23T18:12:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-23T18:12:38.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:12:38.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:38.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:38.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:38 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:38.975 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:38.975 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi110 2024-04-23T18:12:38.975 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-23T18:12:38.975 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:12:38.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:38.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:38.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:38 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:39.877 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:39 smithi131 ceph-mon[33041]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:39.877 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:39 smithi131 ceph-mon[33041]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:12:39.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:39 smithi110 ceph-mon[32585]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:39.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:39 smithi110 ceph-mon[32585]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:12:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:39 smithi026 ceph-mon[29364]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:39 smithi026 ceph-mon[29364]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:12:40.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:40 smithi110 ceph-mon[32585]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 405 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:40.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:40 smithi110 ceph-mon[32585]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:12:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:40 smithi026 ceph-mon[29364]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 405 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:40 smithi026 ceph-mon[29364]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:12:40.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:40 smithi131 ceph-mon[33041]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 405 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:40.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:40 smithi131 ceph-mon[33041]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:12:41.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/3380927250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]': finished 2024-04-23T18:12:41.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:12:41.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:41.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:41 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/675643066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/3380927250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]': finished 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:41.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:41 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/675643066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:41.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/3380927250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]: dispatch 2024-04-23T18:12:41.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05d9ea6-d3bb-4333-a68a-f2d689e41310"}]': finished 2024-04-23T18:12:41.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:12:41.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:41.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:41 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/675643066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:42 smithi110 ceph-mon[32585]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:42.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:42 smithi026 ceph-mon[29364]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:42.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:42 smithi131 ceph-mon[33041]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 404 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:44 smithi110 ceph-mon[32585]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:44.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:44 smithi026 ceph-mon[29364]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:44.989 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:44 smithi131 ceph-mon[33041]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:46.896 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:46 smithi131 ceph-mon[33041]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:46.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:46 smithi110 ceph-mon[32585]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:46 smithi026 ceph-mon[29364]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:48.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:48 smithi131 ceph-mon[33041]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:48.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:12:48.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:48.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:48 smithi110 ceph-mon[32585]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:48.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:12:48.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:48 smithi026 ceph-mon[29364]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:12:48.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:49.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:49 smithi110 ceph-mon[32585]: Deploying daemon osd.8 on smithi131 2024-04-23T18:12:49.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:49 smithi026 ceph-mon[29364]: Deploying daemon osd.8 on smithi131 2024-04-23T18:12:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:49 smithi131 ceph-mon[33041]: Deploying daemon osd.8 on smithi131 2024-04-23T18:12:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:50 smithi110 ceph-mon[32585]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:50.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:50 smithi026 ceph-mon[29364]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:50 smithi131 ceph-mon[33041]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:52 smithi110 ceph-mon[32585]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:52.951 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:52 smithi131 ceph-mon[33041]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:52.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:52 smithi026 ceph-mon[29364]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:54.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:54 smithi110 ceph-mon[32585]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:54.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:54 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:54 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:54 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:54.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:54 smithi026 ceph-mon[29364]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:54.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:54 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:54 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:54 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:54.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:54 smithi131 ceph-mon[33041]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:54.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:54 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:54 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:54.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:54 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:56.308 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:56 smithi131 ceph-mon[33041]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:56.398 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.402+0000 7f3cad7fa700 1 -- 172.21.15.131:0/2115513687 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3cb0062e80 con 0x7f3c9c0a3cc0 2024-04-23T18:12:56.398 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 8 on host 'smithi131' 2024-04-23T18:12:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 msgr2=0x7f3c9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 0x7f3c9c0a6180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c98005950 tx=0x7f3c980058e0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 msgr2=0x7f3cb0197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000b530 tx=0x7f3ca000b8f0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 shutdown_connections 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3c9c0a3cc0 0x7f3c9c0a6180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cb0103560 0x7f3cb019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cb01019c0 0x7f3cb0197ae0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 --2- 172.21.15.131:0/2115513687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cb0102bc0 0x7f3cb0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 >> 172.21.15.131:0/2115513687 conn(0x7f3cb00fcfa0 msgr2=0x7f3cb0105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 shutdown_connections 2024-04-23T18:12:56.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:12:56.405+0000 7f3cb5deb700 1 -- 172.21.15.131:0/2115513687 wait complete. 2024-04-23T18:12:56.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:56 smithi110 ceph-mon[32585]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:56.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:56 smithi026 ceph-mon[29364]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:56.972 DEBUG:teuthology.orchestra.run.smithi131:osd.8> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T18:12:56.974 INFO:tasks.cephadm:Deploying osd.9 on smithi026 with /dev/vg_nvme/lv_1... 2024-04-23T18:12:56.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:12:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:12:57.242 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:12:57.015+0000 7fe1c855a7c0 -1 osd.8 0 log_to_monitors true 2024-04-23T18:12:57.670 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.671 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.671 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.671 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.671 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.671 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:57 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:57 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:12:57.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:57.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:57 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:12:58.601 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:12:58.376+0000 7fe1be3c0700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T18:12:58.601 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:12:58.387+0000 7fe1b99b2700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:58.601 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:58.601 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:12:58.601 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:12:58.602 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:58.602 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:58.602 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:58 smithi131 ceph-mon[33041]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:58.654 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:58.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:12:58.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:12:58.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:58.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:58.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:58 smithi026 ceph-mon[29364]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:58.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:12:58.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:12:58.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:12:58.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:58.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: from='osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:58.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:58 smithi110 ceph-mon[32585]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00873898 s, 1.2 GB/s 2024-04-23T18:12:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-04-23T18:12:59.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:59 smithi110 ceph-mon[32585]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:12:59.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:59 smithi110 ceph-mon[32585]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:12:59.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:59.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:12:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:59.698 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:59 smithi131 ceph-mon[33041]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:12:59.698 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:59 smithi131 ceph-mon[33041]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:12:59.698 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:59.698 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:12:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:59.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:59 smithi026 ceph-mon[29364]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:12:59.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:59 smithi026 ceph-mon[29364]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:12:59.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:12:59.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:12:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi026:vg_nvme/lv_1 2024-04-23T18:13:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:00.629 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:13:00.629 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:13:00.629 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] boot 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:00.630 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:00 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:00.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] boot 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:00.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:00 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:00.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: osd.8 [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] boot 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:00.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:00 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi131 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 5696M 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:01.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:01.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi131 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 5696M 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:01.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:01.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:01.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:01.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi131 2024-04-23T18:13:01.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 5696M 2024-04-23T18:13:01.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:01.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:13:01.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:01.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:01.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:01.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:02.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 -- 172.21.15.26:0/3550513459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0104660 msgr2=0x7fd9a010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 --2- 172.21.15.26:0/3550513459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0104660 0x7fd9a010af10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd99800b0d0 tx=0x7fd99800b3e0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 -- 172.21.15.26:0/3550513459 shutdown_connections 2024-04-23T18:13:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 --2- 172.21.15.26:0/3550513459 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0104660 0x7fd9a010af10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 --2- 172.21.15.26:0/3550513459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 --2- 172.21.15.26:0/3550513459 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0103cc0 0x7fd9a0104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 -- 172.21.15.26:0/3550513459 >> 172.21.15.26:0/3550513459 conn(0x7fd9a00fe060 msgr2=0x7fd9a01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 -- 172.21.15.26:0/3550513459 shutdown_connections 2024-04-23T18:13:02.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.053+0000 7fd9a500f700 1 -- 172.21.15.26:0/3550513459 wait complete. 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.054+0000 7fd9a500f700 1 Processor -- start 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.054+0000 7fd9a500f700 1 -- start start 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0103cc0 0x7fd9a0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0104660 0x7fd9a01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a010e5d0 con 0x7fd9a0102ac0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a010e450 con 0x7fd9a0103cc0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd9a500f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a010e750 con 0x7fd9a0104660 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50932/0 (socket says 172.21.15.26:50932) 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 -- 172.21.15.26:0/2267338450 learned_addr learned my addr 172.21.15.26:0/2267338450 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99e59c700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0103cc0 0x7fd9a0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99f59e700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0104660 0x7fd9a01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 -- 172.21.15.26:0/2267338450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0104660 msgr2=0x7fd9a01a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0104660 0x7fd9a01a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 -- 172.21.15.26:0/2267338450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0103cc0 msgr2=0x7fd9a0199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0103cc0 0x7fd9a0199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.055+0000 7fd99ed9d700 1 -- 172.21.15.26:0/2267338450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a01a0820 con 0x7fd9a0102ac0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd99ed9d700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd99000b530 tx=0x7fd99000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990015020 con 0x7fd9a0102ac0 2024-04-23T18:13:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd990011ce0 con 0x7fd9a0102ac0 2024-04-23T18:13:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a01a0b10 con 0x7fd9a0102ac0 2024-04-23T18:13:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990010c60 con 0x7fd9a0102ac0 2024-04-23T18:13:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.056+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01a1050 con 0x7fd9a0102ac0 2024-04-23T18:13:02.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.057+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a0069730 con 0x7fd9a0102ac0 2024-04-23T18:13:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.058+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd990018020 con 0x7fd9a0102ac0 2024-04-23T18:13:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.058+0000 7fd987fff700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 0x7fd9880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.059+0000 7fd99e59c700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 0x7fd9880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.059+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fd9900cf170 con 0x7fd9a0102ac0 2024-04-23T18:13:02.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.059+0000 7fd99e59c700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 0x7fd9880a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd994005950 tx=0x7fd99400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:02.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.063+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd99001f020 con 0x7fd9a0102ac0 2024-04-23T18:13:02.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:02.282+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd9a0062e80 con 0x7fd9880a3c40 2024-04-23T18:13:02.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:02.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:13:02.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:13:02.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:02.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:02.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:02.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:02.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:03.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:03 smithi026 ceph-mon[29364]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:03.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:03 smithi026 ceph-mon[29364]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:13:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:03 smithi110 ceph-mon[32585]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:03 smithi110 ceph-mon[32585]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:13:03.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:03 smithi131 ceph-mon[33041]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:03.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:03 smithi131 ceph-mon[33041]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:13:04.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:04 smithi110 ceph-mon[32585]: pgmap v200: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-04-23T18:13:04.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:04 smithi026 ceph-mon[29364]: pgmap v200: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-04-23T18:13:04.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:04 smithi131 ceph-mon[33041]: pgmap v200: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-04-23T18:13:05.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:05 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]: dispatch 2024-04-23T18:13:05.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:05 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]': finished 2024-04-23T18:13:05.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:05 smithi110 ceph-mon[32585]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:13:05.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:05.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]: dispatch 2024-04-23T18:13:05.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]': finished 2024-04-23T18:13:05.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:05 smithi026 ceph-mon[29364]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:13:05.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:05.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:05 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]: dispatch 2024-04-23T18:13:05.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:05 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/963357216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34848530-043d-4447-a445-4c1e4d5f1c8b"}]': finished 2024-04-23T18:13:05.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:05 smithi131 ceph-mon[33041]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:13:05.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:06.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:06 smithi110 ceph-mon[32585]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-23T18:13:06.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:06 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2415640713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:06 smithi026 ceph-mon[29364]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-23T18:13:06.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:06 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2415640713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:06 smithi131 ceph-mon[33041]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 289 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-23T18:13:06.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:06 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2415640713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:08.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:08 smithi110 ceph-mon[32585]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:13:08.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:08 smithi026 ceph-mon[29364]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:13:08.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:08 smithi131 ceph-mon[33041]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-23T18:13:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:10 smithi110 ceph-mon[32585]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:10 smithi026 ceph-mon[29364]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:10.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:10 smithi131 ceph-mon[33041]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:12 smithi110 ceph-mon[32585]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:12.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:12 smithi026 ceph-mon[29364]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:12.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:12 smithi131 ceph-mon[33041]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:13:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:13:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:13:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:14 smithi110 ceph-mon[32585]: Deploying daemon osd.9 on smithi026 2024-04-23T18:13:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:14 smithi110 ceph-mon[32585]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:14.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:14 smithi026 ceph-mon[29364]: Deploying daemon osd.9 on smithi026 2024-04-23T18:13:14.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:14 smithi026 ceph-mon[29364]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:14 smithi131 ceph-mon[33041]: Deploying daemon osd.9 on smithi026 2024-04-23T18:13:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:14 smithi131 ceph-mon[33041]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:16.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:16 smithi026 ceph-mon[29364]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:16.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:16 smithi110 ceph-mon[32585]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:16 smithi131 ceph-mon[33041]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:18 smithi110 ceph-mon[32585]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:18.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:18 smithi026 ceph-mon[29364]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:18 smithi131 ceph-mon[33041]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:19.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:19.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:19 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:19.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:19.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:19 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:19.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:19.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:19.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:19 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:20.893 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:20 smithi026 ceph-mon[29364]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:20.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:20 smithi110 ceph-mon[32585]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:20.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:20 smithi131 ceph-mon[33041]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.645+0000 7fd987fff700 1 -- 172.21.15.26:0/2267338450 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd9a0062e80 con 0x7fd9880a3c40 2024-04-23T18:13:22.641 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 9 on host 'smithi026' 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 msgr2=0x7fd9880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 0x7fd9880a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd994005950 tx=0x7fd99400a300 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 msgr2=0x7fd9a0198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd99000b530 tx=0x7fd99000b840 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 shutdown_connections 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd9880a3c40 0x7fd9880a6100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a0104660 0x7fd9a01a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd9a0102ac0 0x7fd9a0198bc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 --2- 172.21.15.26:0/2267338450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9a0103cc0 0x7fd9a0199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 >> 172.21.15.26:0/2267338450 conn(0x7fd9a00fe060 msgr2=0x7fd9a0106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 shutdown_connections 2024-04-23T18:13:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:13:22.648+0000 7fd9a500f700 1 -- 172.21.15.26:0/2267338450 wait complete. 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:13:22.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:22 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:22.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:22.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:22.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:22 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:23.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:23.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:23.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:23.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:22 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:23.414 DEBUG:teuthology.orchestra.run.smithi026:osd.9> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T18:13:23.417 INFO:tasks.cephadm:Deploying osd.10 on smithi110 with /dev/vg_nvme/lv_1... 2024-04-23T18:13:23.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:13:23.661 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:13:23.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:23 smithi131 ceph-mon[33041]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:13:23.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:23 smithi131 ceph-mon[33041]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:13:23.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:23 smithi131 ceph-mon[33041]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:13:23.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.011 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:23 smithi110 ceph-mon[32585]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:13:24.012 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:23 smithi110 ceph-mon[32585]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:13:24.012 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:23 smithi110 ceph-mon[32585]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:13:24.012 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:23 smithi026 ceph-mon[29364]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:13:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:23 smithi026 ceph-mon[29364]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:13:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:23 smithi026 ceph-mon[29364]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-04-23T18:13:24.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.709 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:13:24.447+0000 7fdfdbdba700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T18:13:24.710 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:13:24.461+0000 7fdfd3ba5700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:13:24.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:24.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:13:24.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:13:24.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:24 smithi110 ceph-mon[32585]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:24 smithi131 ceph-mon[33041]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' 2024-04-23T18:13:25.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:25.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-04-23T18:13:25.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:13:25.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:25.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:25.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:24 smithi026 ceph-mon[29364]: from='osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359]' entity='osd.9' 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00851375 s, 1.2 GB/s 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 2024-04-23T18:13:25.498 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-23T18:13:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:13:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:13:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:25.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] boot 2024-04-23T18:13:25.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:13:25.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:25 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:13:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:13:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:25.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] boot 2024-04-23T18:13:25.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:13:25.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:25 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: osd.9 [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] boot 2024-04-23T18:13:26.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:13:26.139 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:25 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:13:26.166 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_1 2024-04-23T18:13:26.411 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:13:27.659 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:13:27.659 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi026 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 4273M 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:27.660 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:27 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi026 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 4273M 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:27.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:27 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:13:27.890 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi026 2024-04-23T18:13:27.890 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 4273M 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:27.891 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:27 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 -- 172.21.15.110:0/2774969305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 msgr2=0x7f3dc81041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 --2- 172.21.15.110:0/2774969305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81041d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009990 tx=0x7f3dbc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 -- 172.21.15.110:0/2774969305 shutdown_connections 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 --2- 172.21.15.110:0/2774969305 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 0x7f3dc810afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 --2- 172.21.15.110:0/2774969305 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc81011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 --2- 172.21.15.110:0/2774969305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81041d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 -- 172.21.15.110:0/2774969305 >> 172.21.15.110:0/2774969305 conn(0x7f3dc80fc6f0 msgr2=0x7f3dc80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 -- 172.21.15.110:0/2774969305 shutdown_connections 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.882+0000 7f3dccceb700 1 -- 172.21.15.110:0/2774969305 wait complete. 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 Processor -- start 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 -- start start 2024-04-23T18:13:27.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 0x7f3dc81a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810e620 con 0x7f3dc8104710 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810e4a0 con 0x7f3dc8103d70 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dccceb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810e7a0 con 0x7f3dc8100d90 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:46340/0 (socket says 172.21.15.110:46340) 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 -- 172.21.15.110:0/1495007843 learned_addr learned my addr 172.21.15.110:0/1495007843 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc659c700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc6d9d700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 0x7f3dc81a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:27.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 -- 172.21.15.110:0/1495007843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 msgr2=0x7f3dc8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc8198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 -- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 msgr2=0x7f3dc81a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 0x7f3dc81a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc5d9b700 1 -- 172.21.15.110:0/1495007843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc81a08e0 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.883+0000 7f3dc659c700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3dc5d9b700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc003790 tx=0x7f3dbc009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc027070 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc81a0b70 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dbc0041c0 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc00f460 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.884+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc81a1020 con 0x7f3dc8103d70 2024-04-23T18:13:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.885+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc8069730 con 0x7f3dc8103d70 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.886+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3dbc02a070 con 0x7f3dc8103d70 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.887+0000 7f3db77fe700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 0x7f3db00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.887+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f3dbc0d0e40 con 0x7f3dc8103d70 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.887+0000 7f3dc659c700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 0x7f3db00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.887+0000 7f3dc659c700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 0x7f3db00a6100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8199b40 tx=0x7f3db8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:27.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:27.890+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3dbc031080 con 0x7f3dc8103d70 2024-04-23T18:13:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:28.097+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3dc8062e80 con 0x7f3db00a3c40 2024-04-23T18:13:28.804 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:28 smithi110 ceph-mon[32585]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:28.805 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:28.805 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:28.805 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:28 smithi026 ceph-mon[29364]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:28 smithi131 ceph-mon[33041]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:29.824 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:29 smithi110 ceph-mon[32585]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:29.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:29 smithi026 ceph-mon[29364]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:29.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:29 smithi131 ceph-mon[33041]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:30.860 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:30 smithi026 ceph-mon[29364]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:30.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:30 smithi110 ceph-mon[32585]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:30 smithi131 ceph-mon[33041]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/4112224101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]': finished 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:31.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:31 smithi110 ceph-mon[32585]: from='client.? 172.21.15.110:0/4291814637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/4112224101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]': finished 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:31.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:31 smithi026 ceph-mon[29364]: from='client.? 172.21.15.110:0/4291814637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/4112224101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]: dispatch 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5bcbd75-0ca4-450e-b735-3bcb342f53a9"}]': finished 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:31.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:31 smithi131 ceph-mon[33041]: from='client.? 172.21.15.110:0/4291814637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:32.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:32 smithi110 ceph-mon[32585]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:32.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:32 smithi026 ceph-mon[29364]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:32.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:32 smithi131 ceph-mon[33041]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:34.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:34 smithi110 ceph-mon[32585]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:34 smithi026 ceph-mon[29364]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:34 smithi131 ceph-mon[33041]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:36.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:36 smithi110 ceph-mon[32585]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:36 smithi026 ceph-mon[29364]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:36.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:36 smithi131 ceph-mon[33041]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:38.877 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:38 smithi110 ceph-mon[32585]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:38.877 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:38 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:13:38.877 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:38 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:38.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:38 smithi026 ceph-mon[29364]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:38.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:38 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:13:38.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:38 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:38 smithi131 ceph-mon[33041]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:38 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:13:38.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:38 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:39.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:39 smithi110 ceph-mon[32585]: Deploying daemon osd.10 on smithi110 2024-04-23T18:13:39.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:39 smithi026 ceph-mon[29364]: Deploying daemon osd.10 on smithi110 2024-04-23T18:13:39.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:39 smithi131 ceph-mon[33041]: Deploying daemon osd.10 on smithi110 2024-04-23T18:13:40.778 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:40 smithi110 ceph-mon[32585]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:40.893 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:40 smithi131 ceph-mon[33041]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:40.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:40 smithi026 ceph-mon[29364]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:42 smithi110 ceph-mon[32585]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:42.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:42 smithi026 ceph-mon[29364]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:42.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:42 smithi131 ceph-mon[33041]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:44.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:44 smithi110 ceph-mon[32585]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:44.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:44 smithi026 ceph-mon[29364]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:44 smithi131 ceph-mon[33041]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:46.664 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:46 smithi110 ceph-mon[32585]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:46.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.714+0000 7f3db77fe700 1 -- 172.21.15.110:0/1495007843 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3dc8062e80 con 0x7f3db00a3c40 2024-04-23T18:13:46.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:46 smithi026 ceph-mon[29364]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:46.709 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 10 on host 'smithi110' 2024-04-23T18:13:46.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 msgr2=0x7f3db00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 0x7f3db00a6100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8199b40 tx=0x7f3db8008040 comp rx=0 tx=0).stop 2024-04-23T18:13:46.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 msgr2=0x7f3dc81991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc003790 tx=0x7f3dbc009580 comp rx=0 tx=0).stop 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 shutdown_connections 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3db00a3c40 0x7f3db00a6100 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3dc8104710 0x7f3dc81a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8100d90 0x7f3dc8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 --2- 172.21.15.110:0/1495007843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dc8103d70 0x7f3dc81991a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 >> 172.21.15.110:0/1495007843 conn(0x7f3dc80fc6f0 msgr2=0x7f3dc81070f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 shutdown_connections 2024-04-23T18:13:46.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-23T18:13:46.717+0000 7f3dccceb700 1 -- 172.21.15.110:0/1495007843 wait complete. 2024-04-23T18:13:46.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:46 smithi131 ceph-mon[33041]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:47.326 DEBUG:teuthology.orchestra.run.smithi110:osd.10> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T18:13:47.329 INFO:tasks.cephadm:Deploying osd.11 on smithi131 with /dev/vg_nvme/lv_1... 2024-04-23T18:13:47.329 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:13:47.573 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:13:47.957 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:47.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:47.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:48.138 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:48.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:48.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:13:48.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:48.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:13:48.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:48.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:48 smithi131 ceph-mon[33041]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:49.066 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-mon[32585]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:49.066 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:13:48.722+0000 7fc3bf74b700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T18:13:49.067 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:13:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:13:48.735+0000 7fc3bb53e700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: from='osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:48 smithi026 ceph-mon[29364]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-23T18:13:49.515 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:13:49.515 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:13:49.515 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-04-23T18:13:49.516 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-04-23T18:13:49.516 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130012 s, 807 MB/s 2024-04-23T18:13:49.516 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-04-23T18:13:49.801 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:49 smithi131 ceph-mon[33041]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:13:49.801 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:49 smithi131 ceph-mon[33041]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:13:49.801 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:49.801 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:49 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.032 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:49 smithi110 ceph-mon[32585]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:13:50.032 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:49 smithi110 ceph-mon[32585]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:13:50.032 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.032 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:49 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.145 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi131:vg_nvme/lv_1 2024-04-23T18:13:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:49 smithi026 ceph-mon[29364]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-23T18:13:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:49 smithi026 ceph-mon[29364]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:13:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:49 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.380 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] boot 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.806 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.807 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] boot 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:51.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:13:51.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:13:51.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: osd.10 [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] boot 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:51.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:51.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:51.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:51.835 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi110 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:51.836 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:51 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 -- 172.21.15.131:0/1328570887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 msgr2=0x7f379410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 --2- 172.21.15.131:0/1328570887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f379410abf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f378c009990 tx=0x7f378c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 -- 172.21.15.131:0/1328570887 shutdown_connections 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 --2- 172.21.15.131:0/1328570887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f379410abf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 --2- 172.21.15.131:0/1328570887 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794103520 0x7f3794105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 --2- 172.21.15.131:0/1328570887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794105e50 0x7f3794108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 -- 172.21.15.131:0/1328570887 >> 172.21.15.131:0/1328570887 conn(0x7f37940fcf30 msgr2=0x7f37940ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 -- 172.21.15.131:0/1328570887 shutdown_connections 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.078+0000 7f379c635700 1 -- 172.21.15.131:0/1328570887 wait complete. 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 Processor -- start 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 -- start start 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794103520 0x7f3794102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794105e50 0x7f37941014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f379410e520 con 0x7f3794105e50 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f379410e3a0 con 0x7f3794103520 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379c635700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f379410e6a0 con 0x7f3794108780 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37772/0 (socket says 172.21.15.131:37772) 2024-04-23T18:13:52.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 -- 172.21.15.131:0/3477560806 learned_addr learned my addr 172.21.15.131:0/3477560806 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f3799bd0700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794105e50 0x7f37941014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379a3d1700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794103520 0x7f3794102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 -- 172.21.15.131:0/3477560806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794103520 msgr2=0x7f3794102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794103520 0x7f3794102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 -- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794105e50 msgr2=0x7f37941014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794105e50 0x7f37941014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 -- 172.21.15.131:0/3477560806 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37941022c0 con 0x7f3794108780 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.079+0000 7f379abd2700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f378c0047b0 tx=0x7f378c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.080+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c027070 con 0x7f3794108780 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.080+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f378c004100 con 0x7f3794108780 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.080+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37941a2ca0 con 0x7f3794108780 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.080+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c022e10 con 0x7f3794108780 2024-04-23T18:13:52.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.080+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37941a30f0 con 0x7f3794108780 2024-04-23T18:13:52.077 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.081+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3794069730 con 0x7f3794108780 2024-04-23T18:13:52.077 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.082+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f378c02a070 con 0x7f3794108780 2024-04-23T18:13:52.077 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.082+0000 7f378b7fe700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 0x7f37800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:52.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.082+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7f378c0d0c70 con 0x7f3794108780 2024-04-23T18:13:52.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.083+0000 7f379a3d1700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 0x7f37800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:52.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.083+0000 7f379a3d1700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 0x7f37800a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37900060b0 tx=0x7f3790007b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:52.081 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.087+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f378c013070 con 0x7f3794108780 2024-04-23T18:13:52.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi110 2024-04-23T18:13:52.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-23T18:13:52.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:52.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:51 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi110 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:51 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:13:52.289 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:13:52.295+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3794062e80 con 0x7f37800a3c40 2024-04-23T18:13:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:52 smithi131 ceph-mon[33041]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:52 smithi131 ceph-mon[33041]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:13:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:52.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:53.094 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:52 smithi110 ceph-mon[32585]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:53.094 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:52 smithi110 ceph-mon[32585]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:13:53.094 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:53.094 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:53.094 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:53.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:52 smithi026 ceph-mon[29364]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:53.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:52 smithi026 ceph-mon[29364]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:13:53.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:53.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:53.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:53 smithi131 ceph-mon[33041]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:53 smithi131 ceph-mon[33041]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:13:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:53 smithi131 ceph-mon[33041]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:54.106 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:53 smithi110 ceph-mon[32585]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:54.106 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:53 smithi110 ceph-mon[32585]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:13:54.106 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:53 smithi110 ceph-mon[32585]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:54.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:53 smithi026 ceph-mon[29364]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:54.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:53 smithi026 ceph-mon[29364]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:13:54.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:53 smithi026 ceph-mon[29364]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:55.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:54 smithi110 ceph-mon[32585]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:13:55.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:54 smithi026 ceph-mon[29364]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:13:55.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:54 smithi131 ceph-mon[33041]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:13:56.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:56.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/3692731012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]': finished 2024-04-23T18:13:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:13:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:13:56.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:55 smithi110 ceph-mon[32585]: from='client.? 172.21.15.131:0/1237873717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/3692731012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]': finished 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:13:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:55 smithi026 ceph-mon[29364]: from='client.? 172.21.15.131:0/1237873717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:56.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/3692731012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]: dispatch 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17072b7-66af-42ed-a40e-7b472fa918f5"}]': finished 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:13:56.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:55 smithi131 ceph-mon[33041]: from='client.? 172.21.15.131:0/1237873717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:58.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:57 smithi110 ceph-mon[32585]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T18:13:58.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:57 smithi026 ceph-mon[29364]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T18:13:58.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:57 smithi131 ceph-mon[33041]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T18:13:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:13:59 smithi131 ceph-mon[33041]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:14:00.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:13:59 smithi110 ceph-mon[32585]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:14:00.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:13:59 smithi026 ceph-mon[29364]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-23T18:14:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:01 smithi110 ceph-mon[32585]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:14:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:01 smithi026 ceph-mon[29364]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:14:02.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:01 smithi131 ceph-mon[33041]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T18:14:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:14:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:02 smithi110 ceph-mon[32585]: Deploying daemon osd.11 on smithi131 2024-04-23T18:14:03.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:14:03.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:03.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:02 smithi026 ceph-mon[29364]: Deploying daemon osd.11 on smithi131 2024-04-23T18:14:03.236 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:14:03.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:03.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:02 smithi131 ceph-mon[33041]: Deploying daemon osd.11 on smithi131 2024-04-23T18:14:04.066 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:03 smithi131 ceph-mon[33041]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-23T18:14:04.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:03 smithi110 ceph-mon[32585]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-23T18:14:04.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:03 smithi026 ceph-mon[29364]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-23T18:14:06.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:05 smithi110 ceph-mon[32585]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:14:06.202 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:05 smithi131 ceph-mon[33041]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:14:06.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:05 smithi026 ceph-mon[29364]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:14:08.096 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:07 smithi131 ceph-mon[33041]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-23T18:14:08.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:07 smithi110 ceph-mon[32585]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-23T18:14:08.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:07 smithi026 ceph-mon[29364]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-23T18:14:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:09 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:09 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:09 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:10.373 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:10 smithi131 ceph-mon[33041]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:10.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:10 smithi110 ceph-mon[32585]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:10 smithi026 ceph-mon[29364]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:11.761 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 11 on host 'smithi131' 2024-04-23T18:14:11.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.766+0000 7f378b7fe700 1 -- 172.21.15.131:0/3477560806 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3794062e80 con 0x7f37800a3c40 2024-04-23T18:14:11.763 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.769+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 msgr2=0x7f37800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.769+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 0x7f37800a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37900060b0 tx=0x7f3790007b80 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.769+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 msgr2=0x7f3794101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.769+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f378c0047b0 tx=0x7f378c005e80 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 shutdown_connections 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f37800a3c40 0x7f37800a6100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3794108780 0x7f3794101a10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3794103520 0x7f3794102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 --2- 172.21.15.131:0/3477560806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3794105e50 0x7f37941014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 >> 172.21.15.131:0/3477560806 conn(0x7f37940fcf30 msgr2=0x7f3794103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 shutdown_connections 2024-04-23T18:14:11.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:11.770+0000 7f379c635700 1 -- 172.21.15.131:0/3477560806 wait complete. 2024-04-23T18:14:12.294 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:12.294 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.294 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.294 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:12.294 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.295 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:12 smithi131 ceph-mon[33041]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.352 DEBUG:teuthology.orchestra.run.smithi131:osd.11> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T18:14:12.354 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T18:14:12.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T18:14:12.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:12.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:12.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:12 smithi026 ceph-mon[29364]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:12.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:12 smithi110 ceph-mon[32585]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:14:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:13.980 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:13 smithi026 ceph-mon[29364]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:14:13.981 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:13 smithi026 ceph-mon[29364]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:14:13.981 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:13 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:13.981 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:13 smithi026 ceph-mon[29364]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:13.981 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:13 smithi026 ceph-mon[29364]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:13 smithi131 ceph-mon[33041]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:14:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:13 smithi131 ceph-mon[33041]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:14:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:13 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:13 smithi131 ceph-mon[33041]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:13 smithi131 ceph-mon[33041]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:13 smithi110 ceph-mon[32585]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:14:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:13 smithi110 ceph-mon[32585]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:14:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:13 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:13 smithi110 ceph-mon[32585]: from='osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:14.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:13 smithi110 ceph-mon[32585]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 -- 172.21.15.26:0/1844871347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 msgr2=0x7f0874102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 --2- 172.21.15.26:0/1844871347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874102ee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f086c00b0d0 tx=0x7f086c00b3e0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 -- 172.21.15.26:0/1844871347 shutdown_connections 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 --2- 172.21.15.26:0/1844871347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874104660 0x7f087410af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 --2- 172.21.15.26:0/1844871347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874102ee0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 --2- 172.21.15.26:0/1844871347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874103cc0 0x7f0874104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 -- 172.21.15.26:0/1844871347 >> 172.21.15.26:0/1844871347 conn(0x7f08740fe060 msgr2=0x7f08741004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 -- 172.21.15.26:0/1844871347 shutdown_connections 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 -- 172.21.15.26:0/1844871347 wait complete. 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.350+0000 7f087b0a6700 1 Processor -- start 2024-04-23T18:14:14.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 -- start start 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874103cc0 0x7f0874199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874104660 0x7f08741a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f087410e530 con 0x7f0874102ac0 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f087410e3b0 con 0x7f0874104660 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f087b0a6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f087410e6b0 con 0x7f0874103cc0 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47646/0 (socket says 172.21.15.26:47646) 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 -- 172.21.15.26:0/529701329 learned_addr learned my addr 172.21.15.26:0/529701329 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0873fff700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874103cc0 0x7f0874199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0879643700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874104660 0x7f08741a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 -- 172.21.15.26:0/529701329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874103cc0 msgr2=0x7f0874199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874103cc0 0x7f0874199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 -- 172.21.15.26:0/529701329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874104660 msgr2=0x7f08741a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874104660 0x7f08741a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 -- 172.21.15.26:0/529701329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08741a08a0 con 0x7f0874102ac0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.351+0000 7f0878e42700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f086c009040 tx=0x7f086c004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.352+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c01f030 con 0x7f0874102ac0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.352+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f086c007ac0 con 0x7f0874102ac0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.352+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08741a0b30 con 0x7f0874102ac0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.352+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c0046c0 con 0x7f0874102ac0 2024-04-23T18:14:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.352+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08741a0fe0 con 0x7f0874102ac0 2024-04-23T18:14:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.353+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0874069730 con 0x7f0874102ac0 2024-04-23T18:14:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.355+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f086c003a00 con 0x7f0874102ac0 2024-04-23T18:14:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.355+0000 7f0871ffb700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 0x7f08600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.355+0000 7f0873fff700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 0x7f08600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.355+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f086c00f020 con 0x7f0874102ac0 2024-04-23T18:14:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.356+0000 7f0873fff700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 0x7f08600a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0864005950 tx=0x7f086400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:14.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.359+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f086c029020 con 0x7f0874102ac0 2024-04-23T18:14:14.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.568+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0874063650 con 0x7f0874102ac0 2024-04-23T18:14:14.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.569+0000 7f0871ffb700 1 -- 172.21.15.26:0/529701329 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f086c09e450 con 0x7f0874102ac0 2024-04-23T18:14:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:14.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 msgr2=0x7f08600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:14.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 0x7f08600a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0864005950 tx=0x7f086400b410 comp rx=0 tx=0).stop 2024-04-23T18:14:14.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 msgr2=0x7f0874198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f086c009040 tx=0x7f086c004d50 comp rx=0 tx=0).stop 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 shutdown_connections 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f08600a3cc0 0x7f08600a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874104660 0x7f08741a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.571+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0874102ac0 0x7f0874198c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.572+0000 7f087b0a6700 1 --2- 172.21.15.26:0/529701329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0874103cc0 0x7f0874199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.572+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 >> 172.21.15.26:0/529701329 conn(0x7f08740fe060 msgr2=0x7f08741070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.572+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 shutdown_connections 2024-04-23T18:14:14.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:14.572+0000 7f087b0a6700 1 -- 172.21.15.26:0/529701329 wait complete. 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/529701329' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:14.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:14 smithi026 ceph-mon[29364]: from='osd.11 ' entity='osd.11' 2024-04-23T18:14:14.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/529701329' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:14.959 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-mon[33041]: from='osd.11 ' entity='osd.11' 2024-04-23T18:14:14.959 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:14:14.624+0000 7f081ae75700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T18:14:14.959 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:14:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:14:14.809+0000 7f0812c60700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:14:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-04-23T18:14:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:14:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/529701329' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:14 smithi110 ceph-mon[32585]: from='osd.11 ' entity='osd.11' 2024-04-23T18:14:15.203 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1713896029,"num_in_osds":12,"osd_in_since":1713896035,"num_remapped_pgs":0} 2024-04-23T18:14:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: purged_snaps scrub starts 2024-04-23T18:14:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: purged_snaps scrub ok 2024-04-23T18:14:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:15 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: purged_snaps scrub starts 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: purged_snaps scrub ok 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:15.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:15 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: purged_snaps scrub starts 2024-04-23T18:14:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: purged_snaps scrub ok 2024-04-23T18:14:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:16.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:15 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:16.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T18:14:16.458 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:16.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:16.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: Detected new or changed devices on smithi131 2024-04-23T18:14:16.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 4272M 2024-04-23T18:14:16.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] boot 2024-04-23T18:14:16.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:14:16.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:16 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: Detected new or changed devices on smithi131 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 4272M 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] boot 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:14:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:16 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:14:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: Detected new or changed devices on smithi131 2024-04-23T18:14:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 4272M 2024-04-23T18:14:17.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: osd.11 [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] boot 2024-04-23T18:14:17.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T18:14:17.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:16 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:14:17.898 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:17 smithi026 ceph-mon[29364]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:14:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:17 smithi131 ceph-mon[33041]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 -- 172.21.15.26:0/1215476061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c41087f0 msgr2=0x7f43c410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 --2- 172.21.15.26:0/1215476061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c41087f0 0x7f43c410ac00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00b0a0 tx=0x7f43bc00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 -- 172.21.15.26:0/1215476061 shutdown_connections 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 --2- 172.21.15.26:0/1215476061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c41087f0 0x7f43c410ac00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 --2- 172.21.15.26:0/1215476061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c4103590 0x7f43c4105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 --2- 172.21.15.26:0/1215476061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4105ec0 0x7f43c41082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 -- 172.21.15.26:0/1215476061 >> 172.21.15.26:0/1215476061 conn(0x7f43c40fcf40 msgr2=0x7f43c40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 -- 172.21.15.26:0/1215476061 shutdown_connections 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 -- 172.21.15.26:0/1215476061 wait complete. 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.107+0000 7f43c946e700 1 Processor -- start 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 -- start start 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 0x7f43c4102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 0x7f43c4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410e580 con 0x7f43c4105ec0 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410e400 con 0x7f43c41087f0 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c946e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410e700 con 0x7f43c4103590 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48628/0 (socket says 172.21.15.26:48628) 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 -- 172.21.15.26:0/2349477697 learned_addr learned my addr 172.21.15.26:0/2349477697 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c2ffd700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 0x7f43c4102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c37fe700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 0x7f43c4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 -- 172.21.15.26:0/2349477697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 msgr2=0x7f43c4102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 0x7f43c4102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 -- 172.21.15.26:0/2349477697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 msgr2=0x7f43c4101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 0x7f43c4101a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c27fc700 1 -- 172.21.15.26:0/2349477697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c4102330 con 0x7f43c4105ec0 2024-04-23T18:14:18.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c2ffd700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 0x7f43c4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:18.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.108+0000 7f43c37fe700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 0x7f43c4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43c27fc700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b530 tx=0x7f43b800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8015020 con 0x7f43c4105ec0 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c41a2d50 con 0x7f43c4105ec0 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43b8011ce0 con 0x7f43c4105ec0 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c41a32b0 con 0x7f43c4105ec0 2024-04-23T18:14:18.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.109+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8010d20 con 0x7f43c4105ec0 2024-04-23T18:14:18.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.110+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c4069730 con 0x7f43c4105ec0 2024-04-23T18:14:18.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.111+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f43b8018020 con 0x7f43c4105ec0 2024-04-23T18:14:18.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.111+0000 7f43abfff700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 0x7f43ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:18.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.112+0000 7f43c2ffd700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 0x7f43ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:18.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.112+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f43b80cfdb0 con 0x7f43c4105ec0 2024-04-23T18:14:18.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.112+0000 7f43c2ffd700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 0x7f43ac0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f43b4005fd0 tx=0x7f43b4005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:18.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.115+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43b801f020 con 0x7f43c4105ec0 2024-04-23T18:14:18.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:17 smithi110 ceph-mon[32585]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:14:18.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.325+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f43c402e000 con 0x7f43c4105ec0 2024-04-23T18:14:18.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.325+0000 7f43abfff700 1 -- 172.21.15.26:0/2349477697 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f43b809cb90 con 0x7f43c4105ec0 2024-04-23T18:14:18.319 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:18.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 msgr2=0x7f43ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 0x7f43ac0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f43b4005fd0 tx=0x7f43b4005e20 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 msgr2=0x7f43c4101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b530 tx=0x7f43b800b8f0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 shutdown_connections 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f43ac0a3c40 0x7f43ac0a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43c41087f0 0x7f43c4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43c4103590 0x7f43c4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 --2- 172.21.15.26:0/2349477697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f43c4105ec0 0x7f43c4101540 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.327+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 >> 172.21.15.26:0/2349477697 conn(0x7f43c40fcf40 msgr2=0x7f43c4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.328+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 shutdown_connections 2024-04-23T18:14:18.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:18.328+0000 7f43c946e700 1 -- 172.21.15.26:0/2349477697 wait complete. 2024-04-23T18:14:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:18 smithi131 ceph-mon[33041]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:18 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2349477697' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:18.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:18 smithi026 ceph-mon[29364]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:18.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:18 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2349477697' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:18.994 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1713896055,"num_in_osds":12,"osd_in_since":1713896035,"num_remapped_pgs":0} 2024-04-23T18:14:18.995 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:14:19.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:18 smithi110 ceph-mon[32585]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:19.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:18 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2349477697' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:14:19.243 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:19.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:19 smithi026 ceph-mon[29364]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:19 smithi110 ceph-mon[32585]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:19 smithi131 ceph-mon[33041]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:20.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3217103803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08103450 msgr2=0x7f6d0810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:20.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3217103803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08103450 0x7f6d0810f8b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000b0a0 tx=0x7f6d0000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3217103803 shutdown_connections 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3217103803 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08103450 0x7f6d0810f8b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3217103803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08108a50 0x7f6d08108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3217103803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08102a90 0x7f6d08102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.968+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3217103803 >> 172.21.15.26:0/3217103803 conn(0x7f6d08079ba0 msgr2=0x7f6d08079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.969+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3217103803 shutdown_connections 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.969+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3217103803 wait complete. 2024-04-23T18:14:20.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.969+0000 7f6d0e2af700 1 Processor -- start 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.969+0000 7f6d0e2af700 1 -- start start 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.969+0000 7f6d0e2af700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08103450 0x7f6d081994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08108a50 0x7f6d0819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08112be0 con 0x7f6d08102a90 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08112a60 con 0x7f6d08103450 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08112d60 con 0x7f6d08108a50 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48644/0 (socket says 172.21.15.26:48644) 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 -- 172.21.15.26:0/3244840380 learned_addr learned my addr 172.21.15.26:0/3244840380 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d077fe700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08103450 0x7f6d081994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0c84c700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08108a50 0x7f6d0819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 -- 172.21.15.26:0/3244840380 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08108a50 msgr2=0x7f6d0819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08108a50 0x7f6d0819d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 -- 172.21.15.26:0/3244840380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08103450 msgr2=0x7f6d081994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08103450 0x7f6d081994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 -- 172.21.15.26:0/3244840380 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0819dfa0 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d07fff700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800c9b0 tx=0x7f6cf800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8007b00 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf8007ca0 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0819e290 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8018660 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.970+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d081a5b10 con 0x7f6d08102a90 2024-04-23T18:14:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.972+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d08069730 con 0x7f6d08102a90 2024-04-23T18:14:20.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.973+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6cf800f450 con 0x7f6d08102a90 2024-04-23T18:14:20.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.973+0000 7f6d057fa700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 0x7f6cf00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:20.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.973+0000 7f6d077fe700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 0x7f6cf00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:20.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.973+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6cf80d0160 con 0x7f6d08102a90 2024-04-23T18:14:20.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.974+0000 7f6d077fe700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 0x7f6cf00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d0807a130 tx=0x7f6cfc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:20.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:20.977+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6cf8014030 con 0x7f6d08102a90 2024-04-23T18:14:21.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.187+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d0819a2f0 con 0x7f6d08102a90 2024-04-23T18:14:21.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.188+0000 7f6d057fa700 1 -- 172.21.15.26:0/3244840380 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7f6cf809d070 con 0x7f6d08102a90 2024-04-23T18:14:21.183 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:21.183 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":82,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:14:16.736226+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 msgr2=0x7f6cf00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 0x7f6cf00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d0807a130 tx=0x7f6cfc00a380 comp rx=0 tx=0).stop 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 msgr2=0x7f6d08198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800c9b0 tx=0x7f6cf800ccc0 comp rx=0 tx=0).stop 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 shutdown_connections 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6cf00a3c40 0x7f6cf00a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d08108a50 0x7f6d0819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d08102a90 0x7f6d08198f90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:21.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 --2- 172.21.15.26:0/3244840380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d08103450 0x7f6d081994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:21.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 >> 172.21.15.26:0/3244840380 conn(0x7f6d08079ba0 msgr2=0x7f6d08101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:21.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 shutdown_connections 2024-04-23T18:14:21.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:21.191+0000 7f6d0e2af700 1 -- 172.21.15.26:0/3244840380 wait complete. 2024-04-23T18:14:21.969 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:21.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T18:14:22.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:21 smithi110 ceph-mon[32585]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:22.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:21 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3244840380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:21 smithi026 ceph-mon[29364]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:22.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:21 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3244840380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:22.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:22.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:21 smithi131 ceph-mon[33041]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:22.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:21 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3244840380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 -- 172.21.15.26:0/4036063487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc107920 msgr2=0x7f4afc107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 --2- 172.21.15.26:0/4036063487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc107920 0x7f4afc107d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009990 tx=0x7f4af0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 -- 172.21.15.26:0/4036063487 shutdown_connections 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 --2- 172.21.15.26:0/4036063487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc102330 0x7f4afc10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 --2- 172.21.15.26:0/4036063487 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc107920 0x7f4afc107d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 --2- 172.21.15.26:0/4036063487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 0x7f4afc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.946+0000 7f4b03148700 1 -- 172.21.15.26:0/4036063487 >> 172.21.15.26:0/4036063487 conn(0x7f4afc0788f0 msgr2=0x7f4afc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- 172.21.15.26:0/4036063487 shutdown_connections 2024-04-23T18:14:23.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- 172.21.15.26:0/4036063487 wait complete. 2024-04-23T18:14:23.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 Processor -- start 2024-04-23T18:14:23.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- start start 2024-04-23T18:14:23.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 0x7f4afc1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:23.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc107920 0x7f4afc103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc111b60 con 0x7f4afc102330 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc1119e0 con 0x7f4afc101970 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4b03148700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc111ce0 con 0x7f4afc107920 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.947+0000 7f4afbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48656/0 (socket says 172.21.15.26:48656) 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 -- 172.21.15.26:0/488398636 learned_addr learned my addr 172.21.15.26:0/488398636 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4b00ee4700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 0x7f4afc1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4b016e5700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc107920 0x7f4afc103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 -- 172.21.15.26:0/488398636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc107920 msgr2=0x7f4afc103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc107920 0x7f4afc103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 -- 172.21.15.26:0/488398636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 msgr2=0x7f4afc1034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:23.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 0x7f4afc1034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 -- 172.21.15.26:0/488398636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc1a7690 con 0x7f4afc102330 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4afbfff700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800d950 tx=0x7f4ae800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8016070 con 0x7f4afc102330 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ae8009df0 con 0x7f4afc102330 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc1a7980 con 0x7f4afc102330 2024-04-23T18:14:23.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.948+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800f3c0 con 0x7f4afc102330 2024-04-23T18:14:23.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.949+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc1a7f10 con 0x7f4afc102330 2024-04-23T18:14:23.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.949+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4afc069730 con 0x7f4afc102330 2024-04-23T18:14:23.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.950+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4ae8019070 con 0x7f4afc102330 2024-04-23T18:14:23.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.951+0000 7f4af9ffb700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 0x7f4aec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:23.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.951+0000 7f4b00ee4700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 0x7f4aec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:23.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.951+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4ae80cfab0 con 0x7f4afc102330 2024-04-23T18:14:23.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.952+0000 7f4b00ee4700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 0x7f4aec0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b5c0 tx=0x7f4af0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:23.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:23.956+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ae809c930 con 0x7f4afc102330 2024-04-23T18:14:23.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:23 smithi026 ceph-mon[29364]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:24.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.160+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4afc02df80 con 0x7f4afc102330 2024-04-23T18:14:24.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.160+0000 7f4af9ffb700 1 -- 172.21.15.26:0/488398636 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f4ae809c230 con 0x7f4afc102330 2024-04-23T18:14:24.154 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:14:24.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 msgr2=0x7f4aec0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:24.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 0x7f4aec0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b5c0 tx=0x7f4af0005960 comp rx=0 tx=0).stop 2024-04-23T18:14:24.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 msgr2=0x7f4afc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800d950 tx=0x7f4ae800dc60 comp rx=0 tx=0).stop 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 shutdown_connections 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4aec0a3cc0 0x7f4aec0a6180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4afc107920 0x7f4afc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4afc101970 0x7f4afc1034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 --2- 172.21.15.26:0/488398636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4afc102330 0x7f4afc103a30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.162+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 >> 172.21.15.26:0/488398636 conn(0x7f4afc0788f0 msgr2=0x7f4afc06b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.163+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 shutdown_connections 2024-04-23T18:14:24.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:24.163+0000 7f4b03148700 1 -- 172.21.15.26:0/488398636 wait complete. 2024-04-23T18:14:24.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:23 smithi110 ceph-mon[32585]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:23 smithi131 ceph-mon[33041]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:24.904 INFO:tasks.cephadm:Adding mds.a on smithi026 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.d on smithi026 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.g on smithi026 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.j on smithi026 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.b on smithi110 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.e on smithi110 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.h on smithi110 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.k on smithi110 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.c on smithi131 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.f on smithi131 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.i on smithi131 2024-04-23T18:14:24.905 INFO:tasks.cephadm:Adding mds.l on smithi131 2024-04-23T18:14:24.906 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l' 2024-04-23T18:14:25.156 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:14:25.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:24 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/488398636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:25.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:24 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/488398636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:25.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:24 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/488398636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:26.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:25 smithi110 ceph-mon[32585]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:26.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:25 smithi026 ceph-mon[29364]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:25 smithi131 ceph-mon[33041]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:26.889 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 -- 172.21.15.131:0/3759865985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4103530 msgr2=0x7f5cf4105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3759865985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4103530 0x7f5cf4105920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009990 tx=0x7f5cf0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 -- 172.21.15.131:0/3759865985 shutdown_connections 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3759865985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4108790 0x7f5cf410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3759865985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4103530 0x7f5cf4105920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3759865985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4105e60 0x7f5cf4108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 -- 172.21.15.131:0/3759865985 >> 172.21.15.131:0/3759865985 conn(0x7f5cf40fcf40 msgr2=0x7f5cf40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 -- 172.21.15.131:0/3759865985 shutdown_connections 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.893+0000 7f5cfc305700 1 -- 172.21.15.131:0/3759865985 wait complete. 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 Processor -- start 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 -- start start 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4103530 0x7f5cf4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:26.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4108790 0x7f5cf41a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf410e5a0 con 0x7f5cf4108790 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf410e420 con 0x7f5cf4103530 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfc305700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf410e720 con 0x7f5cf4105e60 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cf98a0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cf98a0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:44582/0 (socket says 172.21.15.131:44582) 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cf98a0700 1 -- 172.21.15.131:0/3027293001 learned_addr learned my addr 172.21.15.131:0/3027293001 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfa0a1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4103530 0x7f5cf4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.894+0000 7f5cfa8a2700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4108790 0x7f5cf41a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 -- 172.21.15.131:0/3027293001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4103530 msgr2=0x7f5cf4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4103530 0x7f5cf4198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 -- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4108790 msgr2=0x7f5cf41a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4108790 0x7f5cf41a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 -- 172.21.15.131:0/3027293001 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf41a08c0 con 0x7f5cf4105e60 2024-04-23T18:14:26.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cf98a0700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400d950 tx=0x7f5ce400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:26.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4016070 con 0x7f5cf4105e60 2024-04-23T18:14:26.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce400fd40 con 0x7f5cf4105e60 2024-04-23T18:14:26.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf41a0bb0 con 0x7f5cf4105e60 2024-04-23T18:14:26.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400e3b0 con 0x7f5cf4105e60 2024-04-23T18:14:26.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.895+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf41a1140 con 0x7f5cf4105e60 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.896+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf4069730 con 0x7f5cf4105e60 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.897+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5ce4019070 con 0x7f5cf4105e60 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.898+0000 7f5ceb7fe700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 0x7f5ce00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.898+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5ce40cf9b0 con 0x7f5cf4105e60 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.898+0000 7f5cfa0a1700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 0x7f5ce00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:26.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.899+0000 7f5cfa0a1700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 0x7f5ce00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000b5c0 tx=0x7f5cf0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:26.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:26.903+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ce409c830 con 0x7f5cf4105e60 2024-04-23T18:14:27.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.110+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}) -- 0x7f5cf4063140 con 0x7f5ce00a3cc0 2024-04-23T18:14:27.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.118+0000 7f5ceb7fe700 1 -- 172.21.15.131:0/3027293001 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5cf4063140 con 0x7f5ce00a3cc0 2024-04-23T18:14:27.113 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mds.all update... 2024-04-23T18:14:27.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 msgr2=0x7f5ce00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:27.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 0x7f5ce00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000b5c0 tx=0x7f5cf0005fb0 comp rx=0 tx=0).stop 2024-04-23T18:14:27.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 msgr2=0x7f5cf41991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:27.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400d950 tx=0x7f5ce400dc60 comp rx=0 tx=0).stop 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 shutdown_connections 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ce00a3cc0 0x7f5ce00a6180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5cf4108790 0x7f5cf41a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf4103530 0x7f5cf4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 --2- 172.21.15.131:0/3027293001 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5cf4105e60 0x7f5cf41991b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 >> 172.21.15.131:0/3027293001 conn(0x7f5cf40fcf40 msgr2=0x7f5cf4103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 shutdown_connections 2024-04-23T18:14:27.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-04-23T18:14:27.120+0000 7f5cfc305700 1 -- 172.21.15.131:0/3027293001 wait complete. 2024-04-23T18:14:27.690 DEBUG:teuthology.orchestra.run.smithi026:mds.a> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.a.service 2024-04-23T18:14:27.693 DEBUG:teuthology.orchestra.run.smithi026:mds.d> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.d.service 2024-04-23T18:14:27.735 DEBUG:teuthology.orchestra.run.smithi026:mds.g> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.g.service 2024-04-23T18:14:27.778 DEBUG:teuthology.orchestra.run.smithi026:mds.j> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.j.service 2024-04-23T18:14:27.822 DEBUG:teuthology.orchestra.run.smithi110:mds.b> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.b.service 2024-04-23T18:14:27.825 DEBUG:teuthology.orchestra.run.smithi110:mds.e> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.e.service 2024-04-23T18:14:27.867 DEBUG:teuthology.orchestra.run.smithi110:mds.h> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.h.service 2024-04-23T18:14:27.910 DEBUG:teuthology.orchestra.run.smithi110:mds.k> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.k.service 2024-04-23T18:14:27.953 DEBUG:teuthology.orchestra.run.smithi131:mds.c> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c.service 2024-04-23T18:14:27.956 DEBUG:teuthology.orchestra.run.smithi131:mds.f> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.f.service 2024-04-23T18:14:27.999 DEBUG:teuthology.orchestra.run.smithi131:mds.i> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.i.service 2024-04-23T18:14:28.041 DEBUG:teuthology.orchestra.run.smithi131:mds.l> sudo journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.l.service 2024-04-23T18:14:28.084 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-23T18:14:28.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 -- 172.21.15.26:0/1777991254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 msgr2=0x7f2cc810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1777991254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc810a0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009ca0 tx=0x7f2cc002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 -- 172.21.15.26:0/1777991254 shutdown_connections 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1777991254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc810a0e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1777991254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1777991254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 -- 172.21.15.26:0/1777991254 >> 172.21.15.26:0/1777991254 conn(0x7f2cc80fc440 msgr2=0x7f2cc80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 -- 172.21.15.26:0/1777991254 shutdown_connections 2024-04-23T18:14:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 -- 172.21.15.26:0/1777991254 wait complete. 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.205+0000 7f2ccf520640 1 Processor -- start 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 -- start start 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810daf0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810d970 con 0x7f2cc8102a90 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccf520640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810dc70 con 0x7f2cc81053c0 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccd2bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2cccabb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41256/0 (socket says 172.21.15.26:41256) 2024-04-23T18:14:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 -- 172.21.15.26:0/1728701203 learned_addr learned my addr 172.21.15.26:0/1728701203 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccd2bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc81981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:60040/0 (socket says 172.21.15.26:60040) 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 -- 172.21.15.26:0/1728701203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 msgr2=0x7f2cc81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 -- 172.21.15.26:0/1728701203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 msgr2=0x7f2cc81981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc81981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 -- 172.21.15.26:0/1728701203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc819fdc0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2cccabb640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2ccdabd640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00097b0 tx=0x7f2cc0031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.206+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0003bf0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.207+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc0003d90 con 0x7f2cc8107cf0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.207+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc81a0050 con 0x7f2cc8107cf0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.207+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc002fc70 con 0x7f2cc8107cf0 2024-04-23T18:14:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.207+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc81a05f0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.208+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc80fdf30 con 0x7f2cc8107cf0 2024-04-23T18:14:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.209+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2cc0004040 con 0x7f2cc8107cf0 2024-04-23T18:14:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.209+0000 7f2cbe7fc640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 0x7f2cb40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.210+0000 7f2ccd2bc640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 0x7f2cb40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.210+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2cc00f1640 con 0x7f2cc8107cf0 2024-04-23T18:14:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.210+0000 7f2ccd2bc640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 0x7f2cb40a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4005d00 tx=0x7f2cc400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.213+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cc00be4c0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: Saving service mds.all spec with placement smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2024-04-23T18:14:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:28.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:28 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.401+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cc8062b50 con 0x7f2cc8107cf0 2024-04-23T18:14:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.402+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/1728701203 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7f2cc00bddc0 con 0x7f2cc8107cf0 2024-04-23T18:14:28.396 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:28.396 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":82,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:14:16.736226+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 msgr2=0x7f2cb40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 0x7f2cb40a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4005d00 tx=0x7f2cc400b410 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 msgr2=0x7f2cc819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00097b0 tx=0x7f2cc0031840 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 shutdown_connections 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2cb40a3c30 0x7f2cb40a60f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2cc8107cf0 0x7f2cc819f750 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2cc81053c0 0x7f2cc81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 --2- 172.21.15.26:0/1728701203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8102a90 0x7f2cc81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 >> 172.21.15.26:0/1728701203 conn(0x7f2cc80fc440 msgr2=0x7f2cc81033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 shutdown_connections 2024-04-23T18:14:28.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.404+0000 7f2ccf520640 1 -- 172.21.15.26:0/1728701203 wait complete. 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: Saving service mds.all spec with placement smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:28.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:28 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.410 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:14:28.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:14:28.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: Saving service mds.all spec with placement smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:28.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:28 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.520+0000 7f293ab64640 1 -- 172.21.15.26:0/3968526105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 msgr2=0x7f29341092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.520+0000 7f293ab64640 1 --2- 172.21.15.26:0/3968526105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f29341092c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f292c009ca0 tx=0x7f292c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- 172.21.15.26:0/3968526105 shutdown_connections 2024-04-23T18:14:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- 172.21.15.26:0/3968526105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 0x7f2934102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- 172.21.15.26:0/3968526105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f29341092c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- 172.21.15.26:0/3968526105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 0x7f29341012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- 172.21.15.26:0/3968526105 >> 172.21.15.26:0/3968526105 conn(0x7f29340fc400 msgr2=0x7f29340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- 172.21.15.26:0/3968526105 shutdown_connections 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- 172.21.15.26:0/3968526105 wait complete. 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 Processor -- start 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- start start 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 0x7f2934069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 0x7f293419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f293410c9f0 con 0x7f29341029a0 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f293410c870 con 0x7f2934101fe0 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.521+0000 7f293ab64640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f293410cb70 con 0x7f2934100e90 2024-04-23T18:14:28.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41270/0 (socket says 172.21.15.26:41270) 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 -- 172.21.15.26:0/637348024 learned_addr learned my addr 172.21.15.26:0/637348024 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2938900640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 0x7f2934069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2933fff640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 0x7f293419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 -- 172.21.15.26:0/637348024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 msgr2=0x7f2934069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 0x7f2934069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 -- 172.21.15.26:0/637348024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 msgr2=0x7f293419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 0x7f293419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 -- 172.21.15.26:0/637348024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293419fdc0 con 0x7f29341029a0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2939101640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f292c031a20 tx=0x7f292c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c004240 con 0x7f29341029a0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292c003e10 con 0x7f29341029a0 2024-04-23T18:14:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293419fff0 con 0x7f29341029a0 2024-04-23T18:14:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c0305c0 con 0x7f29341029a0 2024-04-23T18:14:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.522+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29341a04b0 con 0x7f29341029a0 2024-04-23T18:14:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.524+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29341081a0 con 0x7f29341029a0 2024-04-23T18:14:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.525+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f292c02f660 con 0x7f29341029a0 2024-04-23T18:14:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.525+0000 7f2931ffb640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 0x7f291c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.525+0000 7f2938900640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 0x7f291c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.525+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f292c0f1770 con 0x7f29341029a0 2024-04-23T18:14:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.526+0000 7f2938900640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 0x7f291c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29240058c0 tx=0x7f292400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.530+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f292c0be5f0 con 0x7f29341029a0 2024-04-23T18:14:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.717+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f293402d810 con 0x7f29341029a0 2024-04-23T18:14:28.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.718+0000 7f2931ffb640 1 -- 172.21.15.26:0/637348024 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f292c0bdef0 con 0x7f29341029a0 2024-04-23T18:14:28.712 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:14:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.719+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 msgr2=0x7f291c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.719+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 0x7f291c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29240058c0 tx=0x7f292400a250 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 msgr2=0x7f293419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f292c031a20 tx=0x7f292c003c20 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 shutdown_connections 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f291c0a3bb0 0x7f291c0a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2934101fe0 0x7f293419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29341029a0 0x7f293419f4b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 --2- 172.21.15.26:0/637348024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2934100e90 0x7f2934069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 >> 172.21.15.26:0/637348024 conn(0x7f29340fc400 msgr2=0x7f2934105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 shutdown_connections 2024-04-23T18:14:28.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.720+0000 7f293ab64640 1 -- 172.21.15.26:0/637348024 wait complete. 2024-04-23T18:14:28.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 -- 172.21.15.26:0/3421515113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 msgr2=0x7f6d9c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 --2- 172.21.15.26:0/3421515113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c10a400 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009ca0 tx=0x7f6d9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 -- 172.21.15.26:0/3421515113 shutdown_connections 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 --2- 172.21.15.26:0/3421515113 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c10a400 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 --2- 172.21.15.26:0/3421515113 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 --2- 172.21.15.26:0/3421515113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c101fc0 0x7f6d9c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 -- 172.21.15.26:0/3421515113 >> 172.21.15.26:0/3421515113 conn(0x7f6d9c0fd550 msgr2=0x7f6d9c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.832+0000 7f6da19fe640 1 -- 172.21.15.26:0/3421515113 shutdown_connections 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 -- 172.21.15.26:0/3421515113 wait complete. 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 Processor -- start 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 -- start start 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c101fc0 0x7f6d9c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10db00 con 0x7f6d9c103b70 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10d980 con 0x7f6d9c1031b0 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6da19fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10dc80 con 0x7f6d9c101fc0 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.833+0000 7f6d9b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41280/0 (socket says 172.21.15.26:41280) 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 -- 172.21.15.26:0/2714108740 learned_addr learned my addr 172.21.15.26:0/2714108740 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9affd640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c101fc0 0x7f6d9c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9a7fc640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 -- 172.21.15.26:0/2714108740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c101fc0 msgr2=0x7f6d9c198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c101fc0 0x7f6d9c198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 -- 172.21.15.26:0/2714108740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 msgr2=0x7f6d9c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 -- 172.21.15.26:0/2714108740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c19fdb0 con 0x7f6d9c103b70 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9a7fc640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6d9b7fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6d940097b0 tx=0x7f6d94003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94004240 con 0x7f6d9c103b70 2024-04-23T18:14:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c1a0040 con 0x7f6d9c103b70 2024-04-23T18:14:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d940317c0 con 0x7f6d9c103b70 2024-04-23T18:14:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9402fd80 con 0x7f6d9c103b70 2024-04-23T18:14:28.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.834+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c1a05e0 con 0x7f6d9c103b70 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.835+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9c1092e0 con 0x7f6d9c103b70 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.836+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d940043e0 con 0x7f6d9c103b70 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.837+0000 7f6da09fc640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 0x7f6d840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.837+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6d940f17d0 con 0x7f6d9c103b70 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.837+0000 7f6d9affd640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 0x7f6d840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.838+0000 7f6d9affd640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 0x7f6d840a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c103010 tx=0x7f6d9000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:28.842+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d940be770 con 0x7f6d9c103b70 2024-04-23T18:14:29.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.068+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d9c062b50 con 0x7f6d9c103b70 2024-04-23T18:14:29.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.068+0000 7f6da09fc640 1 -- 172.21.15.26:0/2714108740 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f6d940be070 con 0x7f6d9c103b70 2024-04-23T18:14:29.062 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:29.063 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-23T18:14:29.063 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 1 2024-04-23T18:14:29.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.070+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 msgr2=0x7f6d840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.070+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 0x7f6d840a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c103010 tx=0x7f6d9000b390 comp rx=0 tx=0).stop 2024-04-23T18:14:29.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.070+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 msgr2=0x7f6d9c19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:29.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.070+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6d940097b0 tx=0x7f6d94003c20 comp rx=0 tx=0).stop 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.070+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 shutdown_connections 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6d840a3bb0 0x7f6d840a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d9c1031b0 0x7f6d9c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6d9c103b70 0x7f6d9c19f740 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 --2- 172.21.15.26:0/2714108740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d9c101fc0 0x7f6d9c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 >> 172.21.15.26:0/2714108740 conn(0x7f6d9c0fd550 msgr2=0x7f6d9c1064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 shutdown_connections 2024-04-23T18:14:29.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:29.071+0000 7f6da19fe640 1 -- 172.21.15.26:0/2714108740 wait complete. 2024-04-23T18:14:29.392 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:29 smithi131 ceph-mon[33041]: Deploying daemon mds.l on smithi131 2024-04-23T18:14:29.392 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:29 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1728701203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:29.392 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:29 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/637348024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:29.392 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:29 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2714108740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:29 smithi110 ceph-mon[32585]: Deploying daemon mds.l on smithi131 2024-04-23T18:14:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:29 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1728701203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:29 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/637348024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:29 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2714108740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:29.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:29 smithi026 ceph-mon[29364]: Deploying daemon mds.l on smithi131 2024-04-23T18:14:29.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:29 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1728701203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:14:29.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:29 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/637348024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:14:29.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:29 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2714108740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:30.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:30 smithi026 ceph-mon[29364]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:30.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:30 smithi110 ceph-mon[32585]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:30.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:30 smithi131 ceph-mon[33041]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:30.493 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:14:30 smithi131 systemd[1]: Started Ceph mds.l for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:31.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:31.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 -- 172.21.15.26:0/1163891176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c107c70 msgr2=0x7fd72c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 --2- 172.21.15.26:0/1163891176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c107c70 0x7fd72c10a0c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd724009ca0 tx=0x7fd72402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 -- 172.21.15.26:0/1163891176 shutdown_connections 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 --2- 172.21.15.26:0/1163891176 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c107c70 0x7fd72c10a0c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 --2- 172.21.15.26:0/1163891176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 0x7fd72c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 --2- 172.21.15.26:0/1163891176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c102a10 0x7fd72c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.188+0000 7fd733768640 1 -- 172.21.15.26:0/1163891176 >> 172.21.15.26:0/1163891176 conn(0x7fd72c0fc400 msgr2=0x7fd72c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- 172.21.15.26:0/1163891176 shutdown_connections 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- 172.21.15.26:0/1163891176 wait complete. 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 Processor -- start 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- start start 2024-04-23T18:14:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 0x7fd72c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c107c70 0x7fd72c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c10dac0 con 0x7fd72c102a10 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c10d940 con 0x7fd72c107c70 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd733768640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c10dc40 con 0x7fd72c105340 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd731504640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd731504640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41302/0 (socket says 172.21.15.26:41302) 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.189+0000 7fd731504640 1 -- 172.21.15.26:0/517518273 learned_addr learned my addr 172.21.15.26:0/517518273 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731d05640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c107c70 0x7fd72c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd730d03640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 0x7fd72c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 -- 172.21.15.26:0/517518273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 msgr2=0x7fd72c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 0x7fd72c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 -- 172.21.15.26:0/517518273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c107c70 msgr2=0x7fd72c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c107c70 0x7fd72c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 -- 172.21.15.26:0/517518273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c19fd90 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd731504640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d950 tx=0x7fd71c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c017070 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71c009c30 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c1a0080 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c00f620 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.190+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1a0620 con 0x7fd72c102a10 2024-04-23T18:14:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.191+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c0fdef0 con 0x7fd72c102a10 2024-04-23T18:14:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.193+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd71c010460 con 0x7fd72c102a10 2024-04-23T18:14:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.194+0000 7fd7227fc640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 0x7fd7180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.194+0000 7fd730d03640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 0x7fd7180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.194+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd71c0cfb90 con 0x7fd72c102a10 2024-04-23T18:14:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.194+0000 7fd730d03640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 0x7fd7180a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd728005fd0 tx=0x7fd728005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.197+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd71c09c9f0 con 0x7fd72c102a10 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6832/2977568812,v1:172.21.15.131:6833/2977568812] up:boot 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: fsmap 1 up:standby 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.401 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:31 smithi110 ceph-mon[32585]: Deploying daemon mds.k on smithi110 2024-04-23T18:14:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.425+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd72c02d5c0 con 0x7fd72c102a10 2024-04-23T18:14:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.426+0000 7fd7227fc640 1 -- 172.21.15.26:0/517518273 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fd71c009de0 con 0x7fd72c102a10 2024-04-23T18:14:31.420 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:31.420 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:14:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 2 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 msgr2=0x7fd7180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 0x7fd7180a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd728005fd0 tx=0x7fd728005850 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 msgr2=0x7fd72c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d950 tx=0x7fd71c00de10 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 shutdown_connections 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd7180a3c30 0x7fd7180a60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd72c107c70 0x7fd72c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd72c105340 0x7fd72c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 --2- 172.21.15.26:0/517518273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd72c102a10 0x7fd72c198170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 >> 172.21.15.26:0/517518273 conn(0x7fd72c0fc400 msgr2=0x7fd72c103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 shutdown_connections 2024-04-23T18:14:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:31.428+0000 7fd733768640 1 -- 172.21.15.26:0/517518273 wait complete. 2024-04-23T18:14:31.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6832/2977568812,v1:172.21.15.131:6833/2977568812] up:boot 2024-04-23T18:14:31.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: fsmap 1 up:standby 2024-04-23T18:14:31.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:14:31.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:31 smithi026 ceph-mon[29364]: Deploying daemon mds.k on smithi110 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6832/2977568812,v1:172.21.15.131:6833/2977568812] up:boot 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: fsmap 1 up:standby 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:31.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:31.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:31.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:31 smithi131 ceph-mon[33041]: Deploying daemon mds.k on smithi110 2024-04-23T18:14:32.648 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:32 smithi110 ceph-mon[32585]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:32.648 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:32 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/517518273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.649 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 systemd[1]: Starting Ceph mds.k for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:32.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:32 smithi026 ceph-mon[29364]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:32.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:32 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/517518273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:32 smithi131 ceph-mon[33041]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:32.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:32 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/517518273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 podman[62074]: 2024-04-23 18:14:32.655002354 +0000 UTC m=+0.113234956 container create 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 podman[62074]: 2024-04-23 18:14:32.55985797 +0000 UTC m=+0.018090529 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 podman[62074]: 2024-04-23 18:14:32.857652204 +0000 UTC m=+0.315884764 container init 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 podman[62074]: 2024-04-23 18:14:32.860276097 +0000 UTC m=+0.318508657 container start 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 ceph-mds[62093]: -- 172.21.15.110:0/1455092794 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581c24aba40 con 0x5581c247e800 2024-04-23T18:14:32.934 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k[62089]: starting mds.k at 2024-04-23T18:14:32.935 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 bash[62074]: 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 2024-04-23T18:14:33.192 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:14:32 smithi110 systemd[1]: Started Ceph mds.k for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:33.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 -- 172.21.15.26:0/433079661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 msgr2=0x7efde81015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 --2- 172.21.15.26:0/433079661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde81015c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efddc009900 tx=0x7efddc02a410 comp rx=0 tx=0).stop 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 -- 172.21.15.26:0/433079661 shutdown_connections 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 --2- 172.21.15.26:0/433079661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 0x7efde8101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 --2- 172.21.15.26:0/433079661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde81015c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 --2- 172.21.15.26:0/433079661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde8100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 -- 172.21.15.26:0/433079661 >> 172.21.15.26:0/433079661 conn(0x7efde80fc400 msgr2=0x7efde80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 -- 172.21.15.26:0/433079661 shutdown_connections 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 -- 172.21.15.26:0/433079661 wait complete. 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.546+0000 7efded69a640 1 Processor -- start 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 -- start start 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 0x7efde81a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7efde810fd20 con 0x7efde8101160 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efde810fba0 con 0x7efde8100800 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efded69a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7efde810fea0 con 0x7efde8101b00 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41312/0 (socket says 172.21.15.26:41312) 2024-04-23T18:14:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 -- 172.21.15.26:0/2068352273 learned_addr learned my addr 172.21.15.26:0/2068352273 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde77fe640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 0x7efde81a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde6ffd640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 -- 172.21.15.26:0/2068352273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 msgr2=0x7efde81a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 0x7efde81a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 -- 172.21.15.26:0/2068352273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 msgr2=0x7efde819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde67fc640 1 -- 172.21.15.26:0/2068352273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde81a2050 con 0x7efde8101160 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.547+0000 7efde6ffd640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efde67fc640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efddc005940 tx=0x7efddc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc03f030 con 0x7efde8101160 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efddc003cb0 con 0x7efde8101160 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc035070 con 0x7efde8101160 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde81a22e0 con 0x7efde8101160 2024-04-23T18:14:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.548+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde81a2850 con 0x7efde8101160 2024-04-23T18:14:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.549+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde810b790 con 0x7efde8101160 2024-04-23T18:14:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.550+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efddc042030 con 0x7efde8101160 2024-04-23T18:14:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.551+0000 7efdcffff640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 0x7efdd00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.551+0000 7efde6ffd640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 0x7efdd00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.551+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efddc0f0b90 con 0x7efde8101160 2024-04-23T18:14:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.552+0000 7efde6ffd640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 0x7efdd00a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006fd0 tx=0x7efdd8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.555+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efddc0bd9b0 con 0x7efde8101160 2024-04-23T18:14:33.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.787+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efde802d5c0 con 0x7efde8101160 2024-04-23T18:14:33.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.787+0000 7efdcffff640 1 -- 172.21.15.26:0/2068352273 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7efddc0b7e20 con 0x7efde8101160 2024-04-23T18:14:33.781 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:33.781 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:14:33.782 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 3 2024-04-23T18:14:33.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 msgr2=0x7efdd00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 0x7efdd00a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006fd0 tx=0x7efdd8008040 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 msgr2=0x7efde819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efddc005940 tx=0x7efddc004430 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 shutdown_connections 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7efdd00a3bb0 0x7efdd00a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efde8101b00 0x7efde81a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efde8101160 0x7efde819a970 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.789+0000 7efded69a640 1 --2- 172.21.15.26:0/2068352273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efde8100800 0x7efde819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.790+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 >> 172.21.15.26:0/2068352273 conn(0x7efde80fc400 msgr2=0x7efde80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.790+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 shutdown_connections 2024-04-23T18:14:33.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:33.790+0000 7efded69a640 1 -- 172.21.15.26:0/2068352273 wait complete. 2024-04-23T18:14:34.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:34.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: Deploying daemon mds.j on smithi026 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6832/1312481724,v1:172.21.15.110:6833/1312481724] up:boot 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: fsmap 2 up:standby 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:14:34.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:33 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2068352273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:34.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: Deploying daemon mds.j on smithi026 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6832/1312481724,v1:172.21.15.110:6833/1312481724] up:boot 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: fsmap 2 up:standby 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:14:34.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:33 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2068352273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:34.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:34.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:34.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:34.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: Deploying daemon mds.j on smithi026 2024-04-23T18:14:34.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6832/1312481724,v1:172.21.15.110:6833/1312481724] up:boot 2024-04-23T18:14:34.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: fsmap 2 up:standby 2024-04-23T18:14:34.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:14:34.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:33 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2068352273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:35.459 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 systemd[1]: Starting Ceph mds.j for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:35.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 podman[67002]: 2024-04-23 18:14:35.452823128 +0000 UTC m=+0.018660577 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 podman[67002]: 2024-04-23 18:14:35.577446804 +0000 UTC m=+0.143284239 container create 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 podman[67002]: 2024-04-23 18:14:35.763679745 +0000 UTC m=+0.329517182 container init 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 podman[67002]: 2024-04-23 18:14:35.76656218 +0000 UTC m=+0.332399621 container start 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mds[67021]: -- 172.21.15.26:0/826125221 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff602f3a40 con 0x55ff602c6c00 2024-04-23T18:14:35.819 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j[67017]: starting mds.j at 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 -- 172.21.15.26:0/111775170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf981009a0 msgr2=0x7fdf9810a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 --2- 172.21.15.26:0/111775170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf981009a0 0x7fdf9810a550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009ca0 tx=0x7fdf9002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 -- 172.21.15.26:0/111775170 shutdown_connections 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 --2- 172.21.15.26:0/111775170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf981009a0 0x7fdf9810a550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 --2- 172.21.15.26:0/111775170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 0x7fdf98100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 --2- 172.21.15.26:0/111775170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf98103570 0x7fdf98103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.910+0000 7fdf9d069640 1 -- 172.21.15.26:0/111775170 >> 172.21.15.26:0/111775170 conn(0x7fdf980fbb80 msgr2=0x7fdf980fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.911+0000 7fdf9d069640 1 -- 172.21.15.26:0/111775170 shutdown_connections 2024-04-23T18:14:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.911+0000 7fdf9d069640 1 -- 172.21.15.26:0/111775170 wait complete. 2024-04-23T18:14:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.911+0000 7fdf9d069640 1 Processor -- start 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 -- start start 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 0x7fdf98198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 0x7fdf98198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810db50 con 0x7fdf98103570 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810d9d0 con 0x7fdf981009a0 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9d069640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810dcd0 con 0x7fdf980fffe0 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9759e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9759e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41348/0 (socket says 172.21.15.26:41348) 2024-04-23T18:14:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 0x7fdf98198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9759e640 1 -- 172.21.15.26:0/3833690808 learned_addr learned my addr 172.21.15.26:0/3833690808 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf96d9d640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 0x7fdf98198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 -- 172.21.15.26:0/3833690808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 msgr2=0x7fdf98198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 0x7fdf98198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 -- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 msgr2=0x7fdf9819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9659c640 1 -- 172.21.15.26:0/3833690808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9819fd70 con 0x7fdf981009a0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.912+0000 7fdf9759e640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf9659c640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 0x7fdf98198690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800d720 tx=0x7fdf8800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88017070 con 0x7fdf981009a0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8800de30 con 0x7fdf981009a0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf981a0060 con 0x7fdf981009a0 2024-04-23T18:14:35.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880047a0 con 0x7fdf981009a0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.913+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf981a0560 con 0x7fdf981009a0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.914+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf98109430 con 0x7fdf981009a0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.915+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf880099d0 con 0x7fdf981009a0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.915+0000 7fdf7ffff640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 0x7fdf800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.916+0000 7fdf96d9d640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 0x7fdf800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.916+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdf880d0360 con 0x7fdf981009a0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.916+0000 7fdf96d9d640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 0x7fdf800a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0095f0 tx=0x7fdf8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:35.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:35.919+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf88020020 con 0x7fdf981009a0 2024-04-23T18:14:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.154+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf9802d5c0 con 0x7fdf981009a0 2024-04-23T18:14:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.155+0000 7fdf7ffff640 1 -- 172.21.15.26:0/3833690808 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fdf8809d180 con 0x7fdf981009a0 2024-04-23T18:14:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:36.149 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:14:36.149 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 4 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 msgr2=0x7fdf800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 0x7fdf800a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0095f0 tx=0x7fdf8c005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 msgr2=0x7fdf98198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 0x7fdf98198690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800d720 tx=0x7fdf8800dbe0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 shutdown_connections 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf800a3bb0 0x7fdf800a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf981009a0 0x7fdf98198690 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf98103570 0x7fdf9819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 --2- 172.21.15.26:0/3833690808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf980fffe0 0x7fdf98198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 >> 172.21.15.26:0/3833690808 conn(0x7fdf980fbb80 msgr2=0x7fdf98106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 shutdown_connections 2024-04-23T18:14:36.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:36.157+0000 7fdf9d069640 1 -- 172.21.15.26:0/3833690808 wait complete. 2024-04-23T18:14:36.161 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 bash[67002]: 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa 2024-04-23T18:14:36.161 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:14:35 smithi026 systemd[1]: Started Ceph mds.j for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:36.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:36.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:35 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:36.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:36.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:36.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:36.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:35 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:36.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:36.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:35 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:37 smithi026 ceph-mon[29364]: Deploying daemon mds.d on smithi026 2024-04-23T18:14:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:37 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.26:6834/1467060531,v1:172.21.15.26:6835/1467060531] up:boot 2024-04-23T18:14:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:37 smithi026 ceph-mon[29364]: fsmap 3 up:standby 2024-04-23T18:14:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:37 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:14:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:37 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3833690808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:37 smithi110 ceph-mon[32585]: Deploying daemon mds.d on smithi026 2024-04-23T18:14:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:37 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.26:6834/1467060531,v1:172.21.15.26:6835/1467060531] up:boot 2024-04-23T18:14:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:37 smithi110 ceph-mon[32585]: fsmap 3 up:standby 2024-04-23T18:14:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:37 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:14:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:37 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3833690808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:37.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:37 smithi131 ceph-mon[33041]: Deploying daemon mds.d on smithi026 2024-04-23T18:14:37.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:37 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.26:6834/1467060531,v1:172.21.15.26:6835/1467060531] up:boot 2024-04-23T18:14:37.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:37 smithi131 ceph-mon[33041]: fsmap 3 up:standby 2024-04-23T18:14:37.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:37 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:14:37.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:37 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3833690808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:38.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:38.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:38 smithi026 ceph-mon[29364]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:38.189 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 systemd[1]: Starting Ceph mds.d for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:38.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:38 smithi131 ceph-mon[33041]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 -- 172.21.15.26:0/3993680703 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 msgr2=0x7f98c01023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 --2- 172.21.15.26:0/3993680703 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c01023e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009900 tx=0x7f98a802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 -- 172.21.15.26:0/3993680703 shutdown_connections 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 --2- 172.21.15.26:0/3993680703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c0103b70 0x7f98c010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 --2- 172.21.15.26:0/3993680703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c01031b0 0x7f98c0103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 --2- 172.21.15.26:0/3993680703 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c01023e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 -- 172.21.15.26:0/3993680703 >> 172.21.15.26:0/3993680703 conn(0x7f98c00fd550 msgr2=0x7f98c00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 -- 172.21.15.26:0/3993680703 shutdown_connections 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.283+0000 7f98c6a83640 1 -- 172.21.15.26:0/3993680703 wait complete. 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 Processor -- start 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 -- start start 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 0x7f98c01987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 0x7f98c019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f98c010db40 con 0x7f98c0101fc0 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f98c010d9c0 con 0x7f98c01031b0 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c6a83640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f98c010dcc0 con 0x7f98c0103b70 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58750/0 (socket says 172.21.15.26:58750) 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 -- 172.21.15.26:0/621823705 learned_addr learned my addr 172.21.15.26:0/621823705 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:38.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c5020640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 0x7f98c019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98bffff640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 0x7f98c01987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 -- 172.21.15.26:0/621823705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 msgr2=0x7f98c019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 0x7f98c019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 -- 172.21.15.26:0/621823705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 msgr2=0x7f98c01987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 0x7f98c01987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 -- 172.21.15.26:0/621823705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c019fea0 con 0x7f98c0101fc0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c5020640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 0x7f98c019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98bffff640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 0x7f98c01987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.284+0000 7f98c481f640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98a8006010 tx=0x7f98a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.285+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a803f030 con 0x7f98c0101fc0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.285+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a8003cb0 con 0x7f98c0101fc0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.285+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c01a0130 con 0x7f98c0101fc0 2024-04-23T18:14:38.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.285+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8035070 con 0x7f98c0101fc0 2024-04-23T18:14:38.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.285+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c01a06a0 con 0x7f98c0101fc0 2024-04-23T18:14:38.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.286+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c0109360 con 0x7f98c0101fc0 2024-04-23T18:14:38.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.287+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f98a8042030 con 0x7f98c0101fc0 2024-04-23T18:14:38.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.287+0000 7f98bdffb640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 0x7f98ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:38.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.288+0000 7f98bffff640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 0x7f98ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:38.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.288+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f98a80f0650 con 0x7f98c0101fc0 2024-04-23T18:14:38.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.288+0000 7f98bffff640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 0x7f98ac0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98b0006fd0 tx=0x7f98b0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:38.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.292+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98a80bd4d0 con 0x7f98c0101fc0 2024-04-23T18:14:38.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:38 smithi110 ceph-mon[32585]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:38.459 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 podman[67361]: 2024-04-23 18:14:38.250227238 +0000 UTC m=+0.018871821 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:38.459 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 podman[67361]: 2024-04-23 18:14:38.37386098 +0000 UTC m=+0.142505551 container create e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:14:38.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.526+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c002d5c0 con 0x7f98c0101fc0 2024-04-23T18:14:38.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.527+0000 7f98bdffb640 1 -- 172.21.15.26:0/621823705 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f98a80bcdd0 con 0x7f98c0101fc0 2024-04-23T18:14:38.521 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:38.521 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:14:38.521 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 4 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 msgr2=0x7f98ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 0x7f98ac0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98b0006fd0 tx=0x7f98b0008040 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 msgr2=0x7f98c0198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98a8006010 tx=0x7f98a80042f0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 shutdown_connections 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f98ac0a3bb0 0x7f98ac0a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98c01031b0 0x7f98c01987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98c0103b70 0x7f98c019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 --2- 172.21.15.26:0/621823705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f98c0101fc0 0x7f98c0198280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 >> 172.21.15.26:0/621823705 conn(0x7f98c00fd550 msgr2=0x7f98c0106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:38.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 shutdown_connections 2024-04-23T18:14:38.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:38.529+0000 7f98c6a83640 1 -- 172.21.15.26:0/621823705 wait complete. 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 podman[67361]: 2024-04-23 18:14:38.626383334 +0000 UTC m=+0.395027903 container init e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 podman[67361]: 2024-04-23 18:14:38.629234389 +0000 UTC m=+0.397878957 container start e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 ceph-mds[67401]: -- 172.21.15.26:0/4059162222 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557216505a40 con 0x5572164d8c00 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d[67395]: starting mds.d at 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 bash[67361]: e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 2024-04-23T18:14:38.959 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:14:38 smithi026 systemd[1]: Started Ceph mds.d for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/621823705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:39.408 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:39 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/621823705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:39 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/621823705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:39 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:40.358 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:40 smithi110 ceph-mon[32585]: Deploying daemon mds.e on smithi110 2024-04-23T18:14:40.358 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:40 smithi110 ceph-mon[32585]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:40.358 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:40 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.26:6836/1500031392,v1:172.21.15.26:6837/1500031392] up:boot 2024-04-23T18:14:40.358 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:40 smithi110 ceph-mon[32585]: fsmap 4 up:standby 2024-04-23T18:14:40.358 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:40 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:14:40.394 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:40 smithi131 ceph-mon[33041]: Deploying daemon mds.e on smithi110 2024-04-23T18:14:40.394 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:40 smithi131 ceph-mon[33041]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:40.394 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:40 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.26:6836/1500031392,v1:172.21.15.26:6837/1500031392] up:boot 2024-04-23T18:14:40.394 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:40 smithi131 ceph-mon[33041]: fsmap 4 up:standby 2024-04-23T18:14:40.394 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:40 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:14:40.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:40 smithi026 ceph-mon[29364]: Deploying daemon mds.e on smithi110 2024-04-23T18:14:40.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:40 smithi026 ceph-mon[29364]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:40.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:40 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.26:6836/1500031392,v1:172.21.15.26:6837/1500031392] up:boot 2024-04-23T18:14:40.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:40 smithi026 ceph-mon[29364]: fsmap 4 up:standby 2024-04-23T18:14:40.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:40 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:14:40.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- 172.21.15.26:0/3542626003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e41029a0 msgr2=0x7ff3e41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/3542626003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e41029a0 0x7ff3e41092c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009ca0 tx=0x7ff3dc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- 172.21.15.26:0/3542626003 shutdown_connections 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/3542626003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/3542626003 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e41029a0 0x7ff3e41092c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/3542626003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4100e90 0x7ff3e41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- 172.21.15.26:0/3542626003 >> 172.21.15.26:0/3542626003 conn(0x7ff3e40fc400 msgr2=0x7ff3e40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- 172.21.15.26:0/3542626003 shutdown_connections 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- 172.21.15.26:0/3542626003 wait complete. 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 Processor -- start 2024-04-23T18:14:40.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 -- start start 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.657+0000 7ff3eaa79640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e41029a0 0x7ff3e419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410c9d0 con 0x7ff3e4100e90 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410c850 con 0x7ff3e41029a0 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410cb50 con 0x7ff3e4101fe0 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58786/0 (socket says 172.21.15.26:58786) 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 -- 172.21.15.26:0/314610182 learned_addr learned my addr 172.21.15.26:0/314610182 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e9016640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e41029a0 0x7ff3e419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e3fff640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 -- 172.21.15.26:0/314610182 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 msgr2=0x7ff3e4197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:40.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 -- 172.21.15.26:0/314610182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e41029a0 msgr2=0x7ff3e419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e41029a0 0x7ff3e419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 -- 172.21.15.26:0/314610182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e419ec60 con 0x7ff3e4100e90 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e3fff640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e8815640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800c920 tx=0x7ff3d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8007c70 con 0x7ff3e4100e90 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3d8007e10 con 0x7ff3e4100e90 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e419ef50 con 0x7ff3e4100e90 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.658+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8018820 con 0x7ff3e4100e90 2024-04-23T18:14:40.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.659+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e419f3b0 con 0x7ff3e4100e90 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.660+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e41080d0 con 0x7ff3e4100e90 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.661+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff3d800f450 con 0x7ff3e4100e90 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.661+0000 7ff3e1ffb640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 0x7ff3d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.662+0000 7ff3e3fff640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 0x7ff3d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.662+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff3d80d0070 con 0x7ff3e4100e90 2024-04-23T18:14:40.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.662+0000 7ff3e3fff640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 0x7ff3d40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4198550 tx=0x7ff3d000b5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:40.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.666+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3d8014030 con 0x7ff3e4100e90 2024-04-23T18:14:40.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.894+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3e402d7b0 con 0x7ff3e4100e90 2024-04-23T18:14:40.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.894+0000 7ff3e1ffb640 1 -- 172.21.15.26:0/314610182 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7ff3d809cef0 con 0x7ff3e4100e90 2024-04-23T18:14:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:40.888 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T18:14:40.889 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 5 2024-04-23T18:14:40.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 msgr2=0x7ff3d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:40.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 0x7ff3d40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4198550 tx=0x7ff3d000b5d0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 msgr2=0x7ff3e4197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800c920 tx=0x7ff3d800cde0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 shutdown_connections 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3d40a3bb0 0x7ff3d40a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3e41029a0 0x7ff3e419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3e4101fe0 0x7ff3e4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 --2- 172.21.15.26:0/314610182 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3e4100e90 0x7ff3e4197040 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 >> 172.21.15.26:0/314610182 conn(0x7ff3e40fc400 msgr2=0x7ff3e41052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 shutdown_connections 2024-04-23T18:14:40.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:40.896+0000 7ff3eaa79640 1 -- 172.21.15.26:0/314610182 wait complete. 2024-04-23T18:14:40.942 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:40 smithi110 systemd[1]: Starting Ceph mds.e for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:41.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:41 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/314610182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:41.303 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:41 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/314610182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:41.303 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 podman[62394]: 2024-04-23 18:14:40.959843759 +0000 UTC m=+0.017891702 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:41.304 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 podman[62394]: 2024-04-23 18:14:41.091648029 +0000 UTC m=+0.149695991 container create 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T18:14:41.304 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 podman[62394]: 2024-04-23 18:14:41.310606293 +0000 UTC m=+0.368654214 container init 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T18:14:41.415 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:41 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/314610182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:41.692 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 podman[62394]: 2024-04-23 18:14:41.31332368 +0000 UTC m=+0.371371609 container start 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True) 2024-04-23T18:14:41.692 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 ceph-mds[62413]: -- 172.21.15.110:0/1926526102 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a5c213a40 con 0x557a5c1e6c00 2024-04-23T18:14:41.692 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: starting mds.e at 2024-04-23T18:14:41.692 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 bash[62394]: 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 2024-04-23T18:14:41.693 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:14:41 smithi110 systemd[1]: Started Ceph mds.e for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:42.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: Deploying daemon mds.f on smithi131 2024-04-23T18:14:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6834/3666168856,v1:172.21.15.110:6835/3666168856] up:boot 2024-04-23T18:14:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: fsmap 5 up:standby 2024-04-23T18:14:42.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:42 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:14:42.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:42.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: Deploying daemon mds.f on smithi131 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6834/3666168856,v1:172.21.15.110:6835/3666168856] up:boot 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: fsmap 5 up:standby 2024-04-23T18:14:42.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:42 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:14:42.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: Deploying daemon mds.f on smithi131 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6834/3666168856,v1:172.21.15.110:6835/3666168856] up:boot 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: fsmap 5 up:standby 2024-04-23T18:14:42.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:42 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 -- 172.21.15.26:0/3039409864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364105360 msgr2=0x7fa364107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 --2- 172.21.15.26:0/3039409864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364105360 0x7fa364107750 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa358009900 tx=0x7fa35802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 -- 172.21.15.26:0/3039409864 shutdown_connections 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 --2- 172.21.15.26:0/3039409864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364107c90 0x7fa36410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 --2- 172.21.15.26:0/3039409864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364105360 0x7fa364107750 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 --2- 172.21.15.26:0/3039409864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364102a30 0x7fa364104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 -- 172.21.15.26:0/3039409864 >> 172.21.15.26:0/3039409864 conn(0x7fa3640fc440 msgr2=0x7fa3640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 -- 172.21.15.26:0/3039409864 shutdown_connections 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.022+0000 7fa368e72640 1 -- 172.21.15.26:0/3039409864 wait complete. 2024-04-23T18:14:43.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 Processor -- start 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 -- start start 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364105360 0x7fa364198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 0x7fa36419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410daf0 con 0x7fa364102a30 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410d970 con 0x7fa364107c90 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa368e72640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa36410dc70 con 0x7fa364105360 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa36259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa36259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58802/0 (socket says 172.21.15.26:58802) 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.023+0000 7fa36259c640 1 -- 172.21.15.26:0/2665222975 learned_addr learned my addr 172.21.15.26:0/2665222975 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa362d9d640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 0x7fa36419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa361d9b640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364105360 0x7fa364198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:43.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 -- 172.21.15.26:0/2665222975 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364105360 msgr2=0x7fa364198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364105360 0x7fa364198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 -- 172.21.15.26:0/2665222975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 msgr2=0x7fa36419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 0x7fa36419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 -- 172.21.15.26:0/2665222975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36419fe60 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa362d9d640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 0x7fa36419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa36259c640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa35400d950 tx=0x7fa35400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354017070 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3641a0150 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa354009c30 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35400f620 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.024+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3641a06c0 con 0x7fa364102a30 2024-04-23T18:14:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.025+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3640fdf30 con 0x7fa364102a30 2024-04-23T18:14:43.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.026+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa354010460 con 0x7fa364102a30 2024-04-23T18:14:43.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.027+0000 7fa3537fe640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 0x7fa34c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:43.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.027+0000 7fa361d9b640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 0x7fa34c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:43.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.027+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa3540cfce0 con 0x7fa364102a30 2024-04-23T18:14:43.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.027+0000 7fa361d9b640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 0x7fa34c0a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa3580314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:43.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.032+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa35409cac0 con 0x7fa364102a30 2024-04-23T18:14:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.259+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa364062b50 con 0x7fa364102a30 2024-04-23T18:14:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.260+0000 7fa3537fe640 1 -- 172.21.15.26:0/2665222975 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7fa354009de0 con 0x7fa364102a30 2024-04-23T18:14:43.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:43.254 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:14:43.254 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 6 2024-04-23T18:14:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 msgr2=0x7fa34c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 0x7fa34c0a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa3580314a0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 msgr2=0x7fa364198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa35400d950 tx=0x7fa35400de10 comp rx=0 tx=0).stop 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 shutdown_connections 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa34c0a3bb0 0x7fa34c0a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa364107c90 0x7fa36419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa364105360 0x7fa364198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 --2- 172.21.15.26:0/2665222975 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa364102a30 0x7fa364198240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.261+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 >> 172.21.15.26:0/2665222975 conn(0x7fa3640fc440 msgr2=0x7fa364103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.262+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 shutdown_connections 2024-04-23T18:14:43.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:43.262+0000 7fa368e72640 1 -- 172.21.15.26:0/2665222975 wait complete. 2024-04-23T18:14:43.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:43 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2665222975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:43.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:43 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2665222975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:43.878 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:14:43 smithi131 ceph-mds[63135]: -- 172.21.15.131:0/2857833010 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3a1e19a40 con 0x55d3a1dec800 2024-04-23T18:14:43.878 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:14:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: starting mds.f at 2024-04-23T18:14:43.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:43 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2665222975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:44.242 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:14:43 smithi131 bash[63116]: e706f4ff5f2b06c85d90afa5eb08525ef16b29b000f9086112547e2c6e13ed82 2024-04-23T18:14:44.242 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:14:43 smithi131 systemd[1]: Started Ceph mds.f for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:44.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:44 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:44.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:44 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:44.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:44.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:44.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:44 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:45.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 -- 172.21.15.26:0/371627139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 msgr2=0x7f866c103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 --2- 172.21.15.26:0/371627139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c103630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8660009900 tx=0x7f866002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 -- 172.21.15.26:0/371627139 shutdown_connections 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 --2- 172.21.15.26:0/371627139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 --2- 172.21.15.26:0/371627139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c103630 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 --2- 172.21.15.26:0/371627139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 0x7f866c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 -- 172.21.15.26:0/371627139 >> 172.21.15.26:0/371627139 conn(0x7f866c0fd550 msgr2=0x7f866c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 -- 172.21.15.26:0/371627139 shutdown_connections 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.389+0000 7f86713e2640 1 -- 172.21.15.26:0/371627139 wait complete. 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 Processor -- start 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 -- start start 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 0x7f866c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f866c10db00 con 0x7f866c1031b0 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f866c10d980 con 0x7f866c101fc0 2024-04-23T18:14:45.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f86713e2640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f866c10dc80 con 0x7f866c103b70 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f866a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f866a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58830/0 (socket says 172.21.15.26:58830) 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f866a7fc640 1 -- 172.21.15.26:0/1217368682 learned_addr learned my addr 172.21.15.26:0/1217368682 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f866b7fe640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.390+0000 7f866affd640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 0x7f866c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 -- 172.21.15.26:0/1217368682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 msgr2=0x7f866c074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 -- 172.21.15.26:0/1217368682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 msgr2=0x7f866c075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 0x7f866c075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 -- 172.21.15.26:0/1217368682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866c075110 con 0x7f866c1031b0 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866b7fe640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:14:45.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f866a7fc640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8660005940 tx=0x7f8660031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866003f030 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866c1a2250 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8660003aa0 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660035070 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.391+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866c1a27a0 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.392+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f866c1092e0 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.393+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8660042030 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.394+0000 7f8653fff640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 0x7f86540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.394+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f86600f04f0 con 0x7f866c1031b0 2024-04-23T18:14:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.394+0000 7f866affd640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 0x7f86540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.395+0000 7f866affd640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 0x7f86540a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f866c103010 tx=0x7f865c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.399+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f86600bcd20 con 0x7f866c1031b0 2024-04-23T18:14:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.627+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f866c02d5c0 con 0x7f866c1031b0 2024-04-23T18:14:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.628+0000 7f8653fff640 1 -- 172.21.15.26:0/1217368682 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f86600c1020 con 0x7f866c1031b0 2024-04-23T18:14:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:45.622 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:14:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 7 2024-04-23T18:14:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 msgr2=0x7f86540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 0x7f86540a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f866c103010 tx=0x7f865c00b390 comp rx=0 tx=0).stop 2024-04-23T18:14:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 msgr2=0x7f866c074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8660005940 tx=0x7f8660031a50 comp rx=0 tx=0).stop 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 shutdown_connections 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f86540a3bb0 0x7f86540a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f866c103b70 0x7f866c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f866c1031b0 0x7f866c074340 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 --2- 172.21.15.26:0/1217368682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f866c101fc0 0x7f866c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 >> 172.21.15.26:0/1217368682 conn(0x7f866c0fd550 msgr2=0x7f866c1064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 shutdown_connections 2024-04-23T18:14:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:45.630+0000 7f86713e2640 1 -- 172.21.15.26:0/1217368682 wait complete. 2024-04-23T18:14:45.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:45 smithi026 ceph-mon[29364]: Deploying daemon mds.c on smithi131 2024-04-23T18:14:45.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:45 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6834/2220145154,v1:172.21.15.131:6835/2220145154] up:boot 2024-04-23T18:14:45.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:45 smithi026 ceph-mon[29364]: fsmap 6 up:standby 2024-04-23T18:14:45.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:45 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:14:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:45 smithi131 ceph-mon[33041]: Deploying daemon mds.c on smithi131 2024-04-23T18:14:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:45 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6834/2220145154,v1:172.21.15.131:6835/2220145154] up:boot 2024-04-23T18:14:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:45 smithi131 ceph-mon[33041]: fsmap 6 up:standby 2024-04-23T18:14:45.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:45 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:14:45.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:45 smithi110 ceph-mon[32585]: Deploying daemon mds.c on smithi131 2024-04-23T18:14:45.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:45 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6834/2220145154,v1:172.21.15.131:6835/2220145154] up:boot 2024-04-23T18:14:45.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:45 smithi110 ceph-mon[32585]: fsmap 6 up:standby 2024-04-23T18:14:45.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:45 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:14:46.462 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 podman[63423]: 2024-04-23 18:14:46.268294802 +0000 UTC m=+0.018157703 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:46.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:46 smithi110 ceph-mon[32585]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:46.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:46 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1217368682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:46.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:46 smithi131 ceph-mon[33041]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:46.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:46 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1217368682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:46.742 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 podman[63423]: 2024-04-23 18:14:46.570463762 +0000 UTC m=+0.320326646 container init cf63d63b7abede0cffb50d3a4bb2b2c5f3600ba0f84e79443d444c6470af040d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:14:46.742 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 podman[63423]: 2024-04-23 18:14:46.573078497 +0000 UTC m=+0.322941382 container start cf63d63b7abede0cffb50d3a4bb2b2c5f3600ba0f84e79443d444c6470af040d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:14:46.742 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 ceph-mds[63443]: -- 172.21.15.131:0/2651768475 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a1055fa40 con 0x561a10533000 2024-04-23T18:14:46.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: starting mds.c at 2024-04-23T18:14:46.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 bash[63423]: cf63d63b7abede0cffb50d3a4bb2b2c5f3600ba0f84e79443d444c6470af040d 2024-04-23T18:14:46.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:14:46 smithi131 systemd[1]: Started Ceph mds.c for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:46 smithi026 ceph-mon[29364]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:46.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:46 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1217368682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:47.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 -- 172.21.15.26:0/1293513575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08105b40 msgr2=0x7fde08105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 --2- 172.21.15.26:0/1293513575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08105b40 0x7fde08105fc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde00009ca0 tx=0x7fde0002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 -- 172.21.15.26:0/1293513575 shutdown_connections 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 --2- 172.21.15.26:0/1293513575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08105b40 0x7fde08105fc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 --2- 172.21.15.26:0/1293513575 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde08103190 0x7fde08105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 --2- 172.21.15.26:0/1293513575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 0x7fde08102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 -- 172.21.15.26:0/1293513575 >> 172.21.15.26:0/1293513575 conn(0x7fde080fc440 msgr2=0x7fde080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 -- 172.21.15.26:0/1293513575 shutdown_connections 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.757+0000 7fde0f751640 1 -- 172.21.15.26:0/1293513575 wait complete. 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 Processor -- start 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 -- start start 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 0x7fde0819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde08105b40 0x7fde081a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fde08114130 con 0x7fde08103190 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde08113fb0 con 0x7fde08100820 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0f751640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fde081142b0 con 0x7fde08105b40 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58852/0 (socket says 172.21.15.26:58852) 2024-04-23T18:14:47.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 -- 172.21.15.26:0/718780625 learned_addr learned my addr 172.21.15.26:0/718780625 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:47.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0d4ed640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 0x7fde0819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:47.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 -- 172.21.15.26:0/718780625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde08105b40 msgr2=0x7fde081a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:47.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde08105b40 0x7fde081a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 -- 172.21.15.26:0/718780625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 msgr2=0x7fde0819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 0x7fde0819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.758+0000 7fde0ccec640 1 -- 172.21.15.26:0/718780625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde081a6370 con 0x7fde08103190 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fde0ccec640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fddf800d770 tx=0x7fddf800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8017070 con 0x7fde08103190 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a6660 con 0x7fde08103190 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf80054d0 con 0x7fde08103190 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8004c30 con 0x7fde08103190 2024-04-23T18:14:47.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.759+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a6bb0 con 0x7fde08103190 2024-04-23T18:14:47.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.760+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde080fdf30 con 0x7fde08103190 2024-04-23T18:14:47.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.762+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fddf8004dd0 con 0x7fde08103190 2024-04-23T18:14:47.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.762+0000 7fddfe7fc640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 0x7fddf40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:47.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.763+0000 7fde0d4ed640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 0x7fddf40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:47.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.763+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fddf80cfdc0 con 0x7fde08103190 2024-04-23T18:14:47.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.763+0000 7fde0d4ed640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 0x7fddf40a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fde04005d00 tx=0x7fde0400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:47.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.766+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fddf809cc40 con 0x7fde08103190 2024-04-23T18:14:47.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: Deploying daemon mds.h on smithi110 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6836/2204987461,v1:172.21.15.131:6837/2204987461] up:boot 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: fsmap 7 up:standby 2024-04-23T18:14:47.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:47 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:14:47.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:47.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: Deploying daemon mds.h on smithi110 2024-04-23T18:14:47.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6836/2204987461,v1:172.21.15.131:6837/2204987461] up:boot 2024-04-23T18:14:47.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: fsmap 7 up:standby 2024-04-23T18:14:47.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:47 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:14:47.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.992+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde0802d770 con 0x7fde08103190 2024-04-23T18:14:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.993+0000 7fddfe7fc640 1 -- 172.21.15.26:0/718780625 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fddf809c540 con 0x7fde08103190 2024-04-23T18:14:47.988 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:47.988 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T18:14:47.988 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 8 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 msgr2=0x7fddf40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 0x7fddf40a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fde04005d00 tx=0x7fde0400b410 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 msgr2=0x7fde0819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fddf800d770 tx=0x7fddf800dc30 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 shutdown_connections 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3bb0 0x7fddf40a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde08105b40 0x7fde081a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08103190 0x7fde0819ec90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 --2- 172.21.15.26:0/718780625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100820 0x7fde0819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 >> 172.21.15.26:0/718780625 conn(0x7fde080fc440 msgr2=0x7fde081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:47.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 shutdown_connections 2024-04-23T18:14:47.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:47.995+0000 7fde0f751640 1 -- 172.21.15.26:0/718780625 wait complete. 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: Deploying daemon mds.h on smithi110 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6836/2204987461,v1:172.21.15.131:6837/2204987461] up:boot 2024-04-23T18:14:47.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: fsmap 7 up:standby 2024-04-23T18:14:47.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:47 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:14:48.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:48 smithi131 ceph-mon[33041]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:48.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:48 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/718780625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:49.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:48 smithi110 ceph-mon[32585]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:49.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:48 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/718780625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:49.121 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:48 smithi110 systemd[1]: Starting Ceph mds.h for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:48 smithi026 ceph-mon[29364]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:49.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:48 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/718780625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:49.391 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 podman[62699]: 2024-04-23 18:14:49.030163468 +0000 UTC m=+0.017828174 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:49.391 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 podman[62699]: 2024-04-23 18:14:49.155339573 +0000 UTC m=+0.143004255 container create 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:14:49.391 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 podman[62699]: 2024-04-23 18:14:49.371898279 +0000 UTC m=+0.359562962 container init 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T18:14:49.391 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 podman[62699]: 2024-04-23 18:14:49.375125055 +0000 UTC m=+0.362789738 container start 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T18:14:49.391 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 ceph-mds[62719]: -- 172.21.15.110:0/2489279723 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d13749a40 con 0x561d1371c800 2024-04-23T18:14:49.692 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: starting mds.h at 2024-04-23T18:14:49.692 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 bash[62699]: 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 2024-04-23T18:14:49.692 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:14:49 smithi110 systemd[1]: Started Ceph mds.h for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:50.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:50.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 -- 172.21.15.26:0/543677914 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec107c90 msgr2=0x7fddec10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:50.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 --2- 172.21.15.26:0/543677914 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec107c90 0x7fddec10a0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009ca0 tx=0x7fdde402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 -- 172.21.15.26:0/543677914 shutdown_connections 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 --2- 172.21.15.26:0/543677914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec105360 0x7fddec107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 --2- 172.21.15.26:0/543677914 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec107c90 0x7fddec10a0e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 --2- 172.21.15.26:0/543677914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec102a30 0x7fddec104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 -- 172.21.15.26:0/543677914 >> 172.21.15.26:0/543677914 conn(0x7fddec0fc440 msgr2=0x7fddec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 -- 172.21.15.26:0/543677914 shutdown_connections 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.122+0000 7fddf23db640 1 -- 172.21.15.26:0/543677914 wait complete. 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 Processor -- start 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 -- start start 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec105360 0x7fddec198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 0x7fddec19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fddec10daf0 con 0x7fddec102a30 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fddec10d970 con 0x7fddec107c90 2024-04-23T18:14:50.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf23db640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fddec10dc70 con 0x7fddec105360 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddebfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddebfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48610/0 (socket says 172.21.15.26:48610) 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddebfff640 1 -- 172.21.15.26:0/3919264170 learned_addr learned my addr 172.21.15.26:0/3919264170 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddf0978640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 0x7fddec19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.123+0000 7fddeb7fe640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec105360 0x7fddec198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 -- 172.21.15.26:0/3919264170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec105360 msgr2=0x7fddec198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec105360 0x7fddec198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 -- 172.21.15.26:0/3919264170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 msgr2=0x7fddec19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 0x7fddec19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 -- 172.21.15.26:0/3919264170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec19fde0 con 0x7fddec102a30 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddf0978640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 0x7fddec19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddebfff640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00e9f0 tx=0x7fdddc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc00cd80 con 0x7fddec102a30 2024-04-23T18:14:50.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdddc0054d0 con 0x7fddec102a30 2024-04-23T18:14:50.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec1a00d0 con 0x7fddec102a30 2024-04-23T18:14:50.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc010690 con 0x7fddec102a30 2024-04-23T18:14:50.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.124+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec1a05d0 con 0x7fddec102a30 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.125+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddec0fdf30 con 0x7fddec102a30 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.126+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdddc004e90 con 0x7fddec102a30 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.127+0000 7fdde97fa640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 0x7fddd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.127+0000 7fddeb7fe640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 0x7fddd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.127+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdddc0d0550 con 0x7fddec102a30 2024-04-23T18:14:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.128+0000 7fddeb7fe640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 0x7fddd40a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddec1996d0 tx=0x7fdde000b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.132+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdddc09d330 con 0x7fddec102a30 2024-04-23T18:14:50.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.357+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddec02d5c0 con 0x7fddec102a30 2024-04-23T18:14:50.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.358+0000 7fdde97fa640 1 -- 172.21.15.26:0/3919264170 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fdddc09cc30 con 0x7fddec102a30 2024-04-23T18:14:50.353 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:50.353 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:14:50.353 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 9 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 msgr2=0x7fddd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 0x7fddd40a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddec1996d0 tx=0x7fdde000b590 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 msgr2=0x7fddec1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00e9f0 tx=0x7fdddc00edf0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 shutdown_connections 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddd40a3bb0 0x7fddd40a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddec107c90 0x7fddec19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddec105360 0x7fddec198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 --2- 172.21.15.26:0/3919264170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddec102a30 0x7fddec1981c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 >> 172.21.15.26:0/3919264170 conn(0x7fddec0fc440 msgr2=0x7fddec103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:50.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 shutdown_connections 2024-04-23T18:14:50.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:50.360+0000 7fddf23db640 1 -- 172.21.15.26:0/3919264170 wait complete. 2024-04-23T18:14:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: Deploying daemon mds.i on smithi131 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:boot 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: fsmap 8 up:standby 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:14:50.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:50 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3919264170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:50.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:50.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: Deploying daemon mds.i on smithi131 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:boot 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: fsmap 8 up:standby 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:14:50.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:50 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3919264170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: Deploying daemon mds.i on smithi131 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:boot 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: fsmap 8 up:standby 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:14:50.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:50 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3919264170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:51.742 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:51 smithi131 systemd[1]: Starting Ceph mds.i for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:52.068 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:51 smithi131 podman[63738]: 2024-04-23 18:14:51.719601867 +0000 UTC m=+0.017958567 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:52.068 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:51 smithi131 podman[63738]: 2024-04-23 18:14:51.83830759 +0000 UTC m=+0.136664272 container create a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T18:14:52.366 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:52.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.475+0000 7f8925f22640 1 -- 172.21.15.26:0/1207595691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920102a10 msgr2=0x7f8920104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.475+0000 7f8925f22640 1 --2- 172.21.15.26:0/1207595691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920102a10 0x7f8920104e00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8910009900 tx=0x7f891002a410 comp rx=0 tx=0).stop 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 -- 172.21.15.26:0/1207595691 shutdown_connections 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 --2- 172.21.15.26:0/1207595691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920107c70 0x7f892010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 --2- 172.21.15.26:0/1207595691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920105340 0x7f8920107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 --2- 172.21.15.26:0/1207595691 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920102a10 0x7f8920104e00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 -- 172.21.15.26:0/1207595691 >> 172.21.15.26:0/1207595691 conn(0x7f89200fc400 msgr2=0x7f89200fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 -- 172.21.15.26:0/1207595691 shutdown_connections 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 -- 172.21.15.26:0/1207595691 wait complete. 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 Processor -- start 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.476+0000 7f8925f22640 1 -- start start 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 0x7f89201981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:52.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920107c70 0x7f892019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f892010dac0 con 0x7f8920105340 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f892010d940 con 0x7f8920102a10 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f892010dc40 con 0x7f8920107c70 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48630/0 (socket says 172.21.15.26:48630) 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 -- 172.21.15.26:0/2446789339 learned_addr learned my addr 172.21.15.26:0/2446789339 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891ffff640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920107c70 0x7f892019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891f7fe640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 0x7f89201981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 -- 172.21.15.26:0/2446789339 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920107c70 msgr2=0x7f892019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920107c70 0x7f892019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 -- 172.21.15.26:0/2446789339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 msgr2=0x7f89201981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 0x7f89201981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 -- 172.21.15.26:0/2446789339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892019fdc0 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891f7fe640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 0x7f89201981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8916ffd640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f890800c970 tx=0x7f890800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908014070 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8908004440 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89201a00b0 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.477+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908003e50 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.478+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89201a0600 con 0x7f8920105340 2024-04-23T18:14:52.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.478+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89200fdef0 con 0x7f8920105340 2024-04-23T18:14:52.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.480+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8908017070 con 0x7f8920105340 2024-04-23T18:14:52.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.480+0000 7f891d7fa640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 0x7f890c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:52.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.481+0000 7f891f7fe640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 0x7f890c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:52.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.481+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f89080d0290 con 0x7f8920105340 2024-04-23T18:14:52.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.481+0000 7f891f7fe640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 0x7f890c0a6210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f89100314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:52.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.484+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f890801e030 con 0x7f8920105340 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 podman[63738]: 2024-04-23 18:14:52.074541777 +0000 UTC m=+0.372898473 container init a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 podman[63738]: 2024-04-23 18:14:52.077285535 +0000 UTC m=+0.375642227 container start a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mds[63757]: -- 172.21.15.131:0/2919411302 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560651941a40 con 0x560651915000 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: starting mds.i at 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 bash[63738]: a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e 2024-04-23T18:14:52.492 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:14:52 smithi131 systemd[1]: Started Ceph mds.i for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:52.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:52 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.717+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f892002d810 con 0x7f8920105340 2024-04-23T18:14:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.717+0000 7f891d7fa640 1 -- 172.21.15.26:0/2446789339 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f890809d0b0 con 0x7f8920105340 2024-04-23T18:14:52.712 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:52.712 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:14:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 10 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 msgr2=0x7f890c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 0x7f890c0a6210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8910000c00 tx=0x7f89100314a0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 msgr2=0x7f89201986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f890800c970 tx=0x7f890800ce30 comp rx=0 tx=0).stop 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 shutdown_connections 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f890c0a3d50 0x7f890c0a6210 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8920107c70 0x7f892019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8920105340 0x7f89201986e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 --2- 172.21.15.26:0/2446789339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8920102a10 0x7f89201981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 >> 172.21.15.26:0/2446789339 conn(0x7f89200fc400 msgr2=0x7f8920103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 shutdown_connections 2024-04-23T18:14:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:52.721+0000 7f8925f22640 1 -- 172.21.15.26:0/2446789339 wait complete. 2024-04-23T18:14:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:52.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:52 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:52 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:53.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:53 smithi026 ceph-mon[29364]: Deploying daemon mds.g on smithi026 2024-04-23T18:14:53.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:boot 2024-04-23T18:14:53.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:53 smithi026 ceph-mon[29364]: fsmap 9 up:standby 2024-04-23T18:14:53.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:53 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:14:53.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:53 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2446789339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:53 smithi110 ceph-mon[32585]: Deploying daemon mds.g on smithi026 2024-04-23T18:14:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:boot 2024-04-23T18:14:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:53 smithi110 ceph-mon[32585]: fsmap 9 up:standby 2024-04-23T18:14:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:53 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:14:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:53 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2446789339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:53 smithi131 ceph-mon[33041]: Deploying daemon mds.g on smithi026 2024-04-23T18:14:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:boot 2024-04-23T18:14:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:53 smithi131 ceph-mon[33041]: fsmap 9 up:standby 2024-04-23T18:14:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:53 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:14:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:53 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2446789339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:54.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:54 smithi026 ceph-mon[29364]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:54.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- 172.21.15.26:0/3482153014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624100880 msgr2=0x7f9624100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 --2- 172.21.15.26:0/3482153014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624100880 0x7f9624100ca0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9620009900 tx=0x7f962002a480 comp rx=0 tx=0).stop 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- 172.21.15.26:0/3482153014 shutdown_connections 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 --2- 172.21.15.26:0/3482153014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9624101b80 0x7f9624102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 --2- 172.21.15.26:0/3482153014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96241011e0 0x7f9624101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 --2- 172.21.15.26:0/3482153014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624100880 0x7f9624100ca0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- 172.21.15.26:0/3482153014 >> 172.21.15.26:0/3482153014 conn(0x7f96240fc400 msgr2=0x7f96240fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- 172.21.15.26:0/3482153014 shutdown_connections 2024-04-23T18:14:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- 172.21.15.26:0/3482153014 wait complete. 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 Processor -- start 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.840+0000 7f962b423640 1 -- start start 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9624100880 0x7f962419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 0x7f962419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f962410fd60 con 0x7f9624101b80 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f962410fbe0 con 0x7f96241011e0 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f962410fee0 con 0x7f9624100880 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48658/0 (socket says 172.21.15.26:48658) 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 -- 172.21.15.26:0/3360093370 learned_addr learned my addr 172.21.15.26:0/3360093370 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96289be640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 0x7f962419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96291bf640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9624100880 0x7f962419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 -- 172.21.15.26:0/3360093370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9624100880 msgr2=0x7f962419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9624100880 0x7f962419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 -- 172.21.15.26:0/3360093370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 msgr2=0x7f962419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 0x7f962419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 -- 172.21.15.26:0/3360093370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96241a2080 con 0x7f9624101b80 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96289be640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 0x7f962419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f96299c0640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f961c00cad0 tx=0x7f961c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c00ae20 con 0x7f9624101b80 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f961c005020 con 0x7f9624101b80 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c00f6b0 con 0x7f9624101b80 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.841+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96241a2370 con 0x7f9624101b80 2024-04-23T18:14:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.842+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96241a28f0 con 0x7f9624101b80 2024-04-23T18:14:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.843+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962410b7a0 con 0x7f9624101b80 2024-04-23T18:14:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.844+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f961c0051d0 con 0x7f9624101b80 2024-04-23T18:14:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.844+0000 7f961a7fc640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 0x7f96100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.844+0000 7f96291bf640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 0x7f96100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.844+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f961c0d00d0 con 0x7f9624101b80 2024-04-23T18:14:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.845+0000 7f96291bf640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 0x7f96100a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9620000c00 tx=0x7f96200314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:54.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:54 smithi110 ceph-mon[32585]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:54.849+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f961c014030 con 0x7f9624101b80 2024-04-23T18:14:54.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:54 smithi131 ceph-mon[33041]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:55.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.081+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9624062b50 con 0x7f9624101b80 2024-04-23T18:14:55.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.082+0000 7f961a7fc640 1 -- 172.21.15.26:0/3360093370 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f961c09cf50 con 0x7f9624101b80 2024-04-23T18:14:55.077 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:55.077 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:14:55.077 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 10 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 msgr2=0x7f96100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 0x7f96100a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9620000c00 tx=0x7f96200314a0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 msgr2=0x7f96241a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f961c00cad0 tx=0x7f961c00cf90 comp rx=0 tx=0).stop 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 shutdown_connections 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f96100a3bb0 0x7f96100a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96241011e0 0x7f962419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9624101b80 0x7f96241a1a10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 --2- 172.21.15.26:0/3360093370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9624100880 0x7f962419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:55.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 >> 172.21.15.26:0/3360093370 conn(0x7f96240fc400 msgr2=0x7f96240fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:55.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 shutdown_connections 2024-04-23T18:14:55.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:55.084+0000 7f962b423640 1 -- 172.21.15.26:0/3360093370 wait complete. 2024-04-23T18:14:55.459 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:14:55 smithi026 systemd[1]: Started Ceph mds.g for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:55.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3360093370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:55.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:55 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3360093370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:55.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:55.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:55 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3360093370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:55.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:55.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:55 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:56.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:56 smithi110 ceph-mon[32585]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:56.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:56 smithi110 ceph-mon[32585]: Deploying daemon mds.a on smithi026 2024-04-23T18:14:56.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:56 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.26:6838/1936694091,v1:172.21.15.26:6839/1936694091] up:boot 2024-04-23T18:14:56.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:56 smithi110 ceph-mon[32585]: fsmap 10 up:standby 2024-04-23T18:14:56.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:56 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:14:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:56 smithi026 ceph-mon[29364]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:56 smithi026 ceph-mon[29364]: Deploying daemon mds.a on smithi026 2024-04-23T18:14:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:56 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.26:6838/1936694091,v1:172.21.15.26:6839/1936694091] up:boot 2024-04-23T18:14:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:56 smithi026 ceph-mon[29364]: fsmap 10 up:standby 2024-04-23T18:14:56.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:56 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:14:56.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:56 smithi131 ceph-mon[33041]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:56.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:56 smithi131 ceph-mon[33041]: Deploying daemon mds.a on smithi026 2024-04-23T18:14:56.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:56 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.26:6838/1936694091,v1:172.21.15.26:6839/1936694091] up:boot 2024-04-23T18:14:56.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:56 smithi131 ceph-mon[33041]: fsmap 10 up:standby 2024-04-23T18:14:56.994 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:56 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:14:57.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:57.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 -- 172.21.15.26:0/1358148611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 msgr2=0x7fd8901012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 --2- 172.21.15.26:0/1358148611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901012b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd884009900 tx=0x7fd88402a410 comp rx=0 tx=0).stop 2024-04-23T18:14:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 -- 172.21.15.26:0/1358148611 shutdown_connections 2024-04-23T18:14:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 --2- 172.21.15.26:0/1358148611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 0x7fd890102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 --2- 172.21.15.26:0/1358148611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 0x7fd8901092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 --2- 172.21.15.26:0/1358148611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901012b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 -- 172.21.15.26:0/1358148611 >> 172.21.15.26:0/1358148611 conn(0x7fd8900fc400 msgr2=0x7fd8900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 -- 172.21.15.26:0/1358148611 shutdown_connections 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.362+0000 7fd897259640 1 -- 172.21.15.26:0/1358148611 wait complete. 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 Processor -- start 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 -- start start 2024-04-23T18:14:57.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 0x7fd890198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 0x7fd89019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010c9e0 con 0x7fd890100e90 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010c860 con 0x7fd890101fe0 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd897259640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010cb60 con 0x7fd8901029a0 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd894ff5640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd894ff5640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48714/0 (socket says 172.21.15.26:48714) 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd894ff5640 1 -- 172.21.15.26:0/531155666 learned_addr learned my addr 172.21.15.26:0/531155666 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.363+0000 7fd88ffff640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 0x7fd890198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd8957f6640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 0x7fd89019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 -- 172.21.15.26:0/531155666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 msgr2=0x7fd89019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 0x7fd89019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 -- 172.21.15.26:0/531155666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 msgr2=0x7fd890198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 0x7fd890198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 -- 172.21.15.26:0/531155666 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89019fea0 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd894ff5640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd884006010 tx=0x7fd8840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88403f030 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd884031910 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884035070 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8901a0130 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.364+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8901a0670 con 0x7fd890100e90 2024-04-23T18:14:57.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.365+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8901a0990 con 0x7fd890100e90 2024-04-23T18:14:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.367+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd884042030 con 0x7fd890100e90 2024-04-23T18:14:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.368+0000 7fd88dffb640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 0x7fd8800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.368+0000 7fd88ffff640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 0x7fd8800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.368+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd8840f0580 con 0x7fd890100e90 2024-04-23T18:14:57.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.369+0000 7fd88ffff640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 0x7fd8800a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0095f0 tx=0x7fd87c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.373+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8840bd360 con 0x7fd890100e90 2024-04-23T18:14:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.622+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8901a0b60 con 0x7fd890100e90 2024-04-23T18:14:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.622+0000 7fd88dffb640 1 -- 172.21.15.26:0/531155666 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7fd8840dbe20 con 0x7fd890100e90 2024-04-23T18:14:57.617 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:57.617 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:14:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 11 2024-04-23T18:14:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 msgr2=0x7fd8800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 0x7fd8800a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0095f0 tx=0x7fd87c005f90 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 msgr2=0x7fd8901982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd884006010 tx=0x7fd8840042f0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 shutdown_connections 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd8800a3c30 0x7fd8800a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd890101fe0 0x7fd890198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8901029a0 0x7fd89019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 --2- 172.21.15.26:0/531155666 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd890100e90 0x7fd8901982d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 >> 172.21.15.26:0/531155666 conn(0x7fd8900fc400 msgr2=0x7fd8900fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 shutdown_connections 2024-04-23T18:14:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:57.626+0000 7fd897259640 1 -- 172.21.15.26:0/531155666 wait complete. 2024-04-23T18:14:57.631 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:57 smithi026 systemd[1]: Starting Ceph mds.a for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:14:57.889 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:57 smithi026 podman[68339]: 2024-04-23 18:14:57.631066423 +0000 UTC m=+0.026391215 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:14:57.889 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:57 smithi026 podman[68339]: 2024-04-23 18:14:57.75285634 +0000 UTC m=+0.148181174 container create eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:57 smithi026 podman[68339]: 2024-04-23 18:14:57.996986642 +0000 UTC m=+0.392311434 container init eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:57 smithi026 podman[68339]: 2024-04-23 18:14:57.99958658 +0000 UTC m=+0.394911376 container start eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mds[68360]: -- 172.21.15.26:0/2912116020 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e715837a40 con 0x55e71580ac00 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a[68356]: starting mds.a at 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:58 smithi026 bash[68339]: eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb 2024-04-23T18:14:58.459 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:14:58 smithi026 systemd[1]: Started Ceph mds.a for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/531155666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:58.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:58 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/531155666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:58.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:58 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/531155666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:14:58.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:58 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:59.633 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 -- 172.21.15.26:0/2188282368 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20107c90 msgr2=0x7f1a2010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 --2- 172.21.15.26:0/2188282368 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20107c90 0x7f1a2010a0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009900 tx=0x7f1a1802a410 comp rx=0 tx=0).stop 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 -- 172.21.15.26:0/2188282368 shutdown_connections 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 --2- 172.21.15.26:0/2188282368 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20107c90 0x7f1a2010a0e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 --2- 172.21.15.26:0/2188282368 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20105360 0x7f1a20107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 --2- 172.21.15.26:0/2188282368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20102a30 0x7f1a20104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 -- 172.21.15.26:0/2188282368 >> 172.21.15.26:0/2188282368 conn(0x7f1a200fc440 msgr2=0x7f1a200fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 -- 172.21.15.26:0/2188282368 shutdown_connections 2024-04-23T18:14:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.756+0000 7f1a25520640 1 -- 172.21.15.26:0/2188282368 wait complete. 2024-04-23T18:14:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 Processor -- start 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 -- start start 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20105360 0x7f1a201009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20107c90 0x7f1a20100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2010db50 con 0x7f1a20102a30 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2010d9d0 con 0x7f1a20105360 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a25520640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2010dcd0 con 0x7f1a20107c90 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35068/0 (socket says 172.21.15.26:35068) 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 -- 172.21.15.26:0/2561343132 learned_addr learned my addr 172.21.15.26:0/2561343132 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1e7fc640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20105360 0x7f1a201009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1f7fe640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20107c90 0x7f1a20100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 -- 172.21.15.26:0/2561343132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20107c90 msgr2=0x7f1a20100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20107c90 0x7f1a20100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 -- 172.21.15.26:0/2561343132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20105360 msgr2=0x7f1a201009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.757+0000 7f1a1effd640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20105360 0x7f1a201009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a1effd640 1 -- 172.21.15.26:0/2561343132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a201017b0 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a1effd640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000cce0 tx=0x7f1a1000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000f450 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a201a2260 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a1000faa0 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10018aa0 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.758+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a201a2830 con 0x7f1a20102a30 2024-04-23T18:14:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.759+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a200fdf30 con 0x7f1a20102a30 2024-04-23T18:14:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.760+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1a10018c80 con 0x7f1a20102a30 2024-04-23T18:14:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.761+0000 7f1a07fff640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 0x7f1a080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.761+0000 7f1a1e7fc640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 0x7f1a080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.761+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1a100d05c0 con 0x7f1a20102a30 2024-04-23T18:14:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.761+0000 7f1a1e7fc640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 0x7f1a080a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1a20102110 tx=0x7f1a14009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.765+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a10014030 con 0x7f1a20102a30 2024-04-23T18:14:59.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:59 smithi110 ceph-mon[32585]: Deploying daemon mds.b on smithi110 2024-04-23T18:14:59.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:59 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.26:6840/2731986985,v1:172.21.15.26:6841/2731986985] up:boot 2024-04-23T18:14:59.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:59 smithi110 ceph-mon[32585]: fsmap 11 up:standby 2024-04-23T18:14:59.924 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:14:59 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:14:59.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:59 smithi026 ceph-mon[29364]: Deploying daemon mds.b on smithi110 2024-04-23T18:14:59.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:59 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.26:6840/2731986985,v1:172.21.15.26:6841/2731986985] up:boot 2024-04-23T18:14:59.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:59 smithi026 ceph-mon[29364]: fsmap 11 up:standby 2024-04-23T18:14:59.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:14:59 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:14:59.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.992+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a2002d5c0 con 0x7f1a20102a30 2024-04-23T18:14:59.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.993+0000 7f1a07fff640 1 -- 172.21.15.26:0/2561343132 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9668 (secure 0 0 0) 0x7f1a1009d3a0 con 0x7f1a20102a30 2024-04-23T18:14:59.988 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:14:59.988 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T18:14:59.988 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 12 2024-04-23T18:14:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 msgr2=0x7f1a080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 0x7f1a080a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1a20102110 tx=0x7f1a14009110 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 msgr2=0x7f1a201023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000cce0 tx=0x7f1a1000a620 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 shutdown_connections 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1a080a3bb0 0x7f1a080a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a20105360 0x7f1a201009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a20107c90 0x7f1a20100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 --2- 172.21.15.26:0/2561343132 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1a20102a30 0x7f1a201023a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 >> 172.21.15.26:0/2561343132 conn(0x7f1a200fc440 msgr2=0x7f1a20103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 shutdown_connections 2024-04-23T18:14:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:14:59.995+0000 7f1a25520640 1 -- 172.21.15.26:0/2561343132 wait complete. 2024-04-23T18:14:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:59 smithi131 ceph-mon[33041]: Deploying daemon mds.b on smithi110 2024-04-23T18:14:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:59 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.26:6840/2731986985,v1:172.21.15.26:6841/2731986985] up:boot 2024-04-23T18:14:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:59 smithi131 ceph-mon[33041]: fsmap 11 up:standby 2024-04-23T18:14:59.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:14:59 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:15:00.174 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 systemd[1]: Starting Ceph mds.b for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:15:00.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:00 smithi110 ceph-mon[32585]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:00 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2561343132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 podman[63021]: 2024-04-23 18:15:00.300032694 +0000 UTC m=+0.018213967 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 podman[63021]: 2024-04-23 18:15:00.405508731 +0000 UTC m=+0.123689996 container create 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 podman[63021]: 2024-04-23 18:15:00.632949998 +0000 UTC m=+0.351131263 container init 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 podman[63021]: 2024-04-23 18:15:00.635515189 +0000 UTC m=+0.353696463 container start 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T18:15:00.652 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 ceph-mds[63040]: -- 172.21.15.110:0/3858679438 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8e2a1ba40 con 0x55a8e29ef000 2024-04-23T18:15:00.942 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: starting mds.b at 2024-04-23T18:15:00.943 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 bash[63021]: 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 2024-04-23T18:15:00.943 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:00 smithi110 systemd[1]: Started Ceph mds.b for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:15:00.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:00 smithi026 ceph-mon[29364]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:00.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:00 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2561343132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:00 smithi131 ceph-mon[33041]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:00.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:00 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2561343132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:01.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:01.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:boot 2024-04-23T18:15:01.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: fsmap 12 up:standby 2024-04-23T18:15:01.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:01 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:02.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:02.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:boot 2024-04-23T18:15:02.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: fsmap 12 up:standby 2024-04-23T18:15:02.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:01 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 -- 172.21.15.26:0/3216764483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678107c90 msgr2=0x7f467810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 --2- 172.21.15.26:0/3216764483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678107c90 0x7f467810a0e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4670009ca0 tx=0x7f467002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 -- 172.21.15.26:0/3216764483 shutdown_connections 2024-04-23T18:15:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 --2- 172.21.15.26:0/3216764483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678105360 0x7f4678107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 --2- 172.21.15.26:0/3216764483 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678107c90 0x7f467810a0e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 --2- 172.21.15.26:0/3216764483 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678102a30 0x7f4678104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 -- 172.21.15.26:0/3216764483 >> 172.21.15.26:0/3216764483 conn(0x7f46780fc440 msgr2=0x7f46780fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 -- 172.21.15.26:0/3216764483 shutdown_connections 2024-04-23T18:15:02.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.112+0000 7f467ed30640 1 -- 172.21.15.26:0/3216764483 wait complete. 2024-04-23T18:15:02.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 Processor -- start 2024-04-23T18:15:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 -- start start 2024-04-23T18:15:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678105360 0x7f4678198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678107c90 0x7f467819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f467810db40 con 0x7f4678102a30 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f467810d9c0 con 0x7f4678107c90 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467ed30640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f467810dcc0 con 0x7f4678105360 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35086/0 (socket says 172.21.15.26:35086) 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 -- 172.21.15.26:0/2247773855 learned_addr learned my addr 172.21.15.26:0/2247773855 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:02.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f4677fff640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678105360 0x7f4678198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467d2cd640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678107c90 0x7f467819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 -- 172.21.15.26:0/2247773855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678105360 msgr2=0x7f4678198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678105360 0x7f4678198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 -- 172.21.15.26:0/2247773855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678107c90 msgr2=0x7f467819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678107c90 0x7f467819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.113+0000 7f467cacc640 1 -- 172.21.15.26:0/2247773855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467819fd90 con 0x7f4678102a30 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f467cacc640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f466c00d950 tx=0x7f466c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c017030 con 0x7f4678102a30 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f466c004020 con 0x7f4678102a30 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46781a0080 con 0x7f4678102a30 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c0047a0 con 0x7f4678102a30 2024-04-23T18:15:02.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.114+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46781a05d0 con 0x7f4678102a30 2024-04-23T18:15:02.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.115+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46780fdf30 con 0x7f4678102a30 2024-04-23T18:15:02.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.117+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f466c009a30 con 0x7f4678102a30 2024-04-23T18:15:02.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.118+0000 7f4675ffb640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 0x7f46680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.118+0000 7f4677fff640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 0x7f46680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.118+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f466c0cfa10 con 0x7f4678102a30 2024-04-23T18:15:02.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.118+0000 7f4677fff640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 0x7f46680a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4678199680 tx=0x7f4664005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.122+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f466c09c7f0 con 0x7f4678102a30 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:02.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:02.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:boot 2024-04-23T18:15:02.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: fsmap 12 up:standby 2024-04-23T18:15:02.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:01 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:15:02.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.382+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f467802d5c0 con 0x7f4678102a30 2024-04-23T18:15:02.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.382+0000 7f4675ffb640 1 -- 172.21.15.26:0/2247773855 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10497 (secure 0 0 0) 0x7f466c09c0f0 con 0x7f4678102a30 2024-04-23T18:15:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:02.377 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-23T18:15:02.378 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 13 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 msgr2=0x7f46680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 0x7f46680a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4678199680 tx=0x7f4664005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 msgr2=0x7f46781981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f466c00d950 tx=0x7f466c00de10 comp rx=0 tx=0).stop 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 shutdown_connections 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f46680a3bb0 0x7f46680a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4678107c90 0x7f467819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4678105360 0x7f4678198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 --2- 172.21.15.26:0/2247773855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4678102a30 0x7f46781981c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 >> 172.21.15.26:0/2247773855 conn(0x7f46780fc440 msgr2=0x7f4678103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 shutdown_connections 2024-04-23T18:15:02.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.385+0000 7f467ed30640 1 -- 172.21.15.26:0/2247773855 wait complete. 2024-04-23T18:15:02.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:02.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 -- 172.21.15.26:0/1614609284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 msgr2=0x7f69c0103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 --2- 172.21.15.26:0/1614609284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0103630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009900 tx=0x7f69b402a410 comp rx=0 tx=0).stop 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 -- 172.21.15.26:0/1614609284 shutdown_connections 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 --2- 172.21.15.26:0/1614609284 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 0x7f69c010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 --2- 172.21.15.26:0/1614609284 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0103630 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 --2- 172.21.15.26:0/1614609284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 0x7f69c01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 -- 172.21.15.26:0/1614609284 >> 172.21.15.26:0/1614609284 conn(0x7f69c00fd550 msgr2=0x7f69c00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 -- 172.21.15.26:0/1614609284 shutdown_connections 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 -- 172.21.15.26:0/1614609284 wait complete. 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.553+0000 7f69c5736640 1 Processor -- start 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 -- start start 2024-04-23T18:15:02.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 0x7f69c0075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 0x7f69c0074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f69c010db00 con 0x7f69c01031b0 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f69c010d980 con 0x7f69c0101fc0 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69c5736640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f69c010dc80 con 0x7f69c0103b70 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35100/0 (socket says 172.21.15.26:35100) 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 -- 172.21.15.26:0/1454785740 learned_addr learned my addr 172.21.15.26:0/1454785740 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69beffd640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 0x7f69c0075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69bf7fe640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 0x7f69c0074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 -- 172.21.15.26:0/1454785740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 msgr2=0x7f69c0074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 0x7f69c0074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 -- 172.21.15.26:0/1454785740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 msgr2=0x7f69c0075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 0x7f69c0075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 -- 172.21.15.26:0/1454785740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c0075110 con 0x7f69c01031b0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.554+0000 7f69be7fc640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f69b4005940 tx=0x7f69b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.555+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b403f030 con 0x7f69c01031b0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.555+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69b4003cb0 con 0x7f69c01031b0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.555+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4035070 con 0x7f69c01031b0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.555+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c01a2290 con 0x7f69c01031b0 2024-04-23T18:15:02.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.555+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c01a2790 con 0x7f69c01031b0 2024-04-23T18:15:02.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.557+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c01092e0 con 0x7f69c01031b0 2024-04-23T18:15:02.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.557+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69b4042030 con 0x7f69c01031b0 2024-04-23T18:15:02.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.558+0000 7f69a7fff640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 0x7f69a80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.558+0000 7f69beffd640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 0x7f69a80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.558+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f69b40f0290 con 0x7f69c01031b0 2024-04-23T18:15:02.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.559+0000 7f69beffd640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 0x7f69a80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69c0103010 tx=0x7f69b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.563+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69b4003e60 con 0x7f69c01031b0 2024-04-23T18:15:02.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.763+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69c002d7c0 con 0x7f69c01031b0 2024-04-23T18:15:02.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.764+0000 7f69a7fff640 1 -- 172.21.15.26:0/1454785740 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7f69b40bcac0 con 0x7f69c01031b0 2024-04-23T18:15:02.758 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:02.758 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":82,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:14:16.736226+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 msgr2=0x7f69a80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 0x7f69a80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69c0103010 tx=0x7f69b0005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 msgr2=0x7f69c0074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f69b4005940 tx=0x7f69b40042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 shutdown_connections 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f69a80a3c30 0x7f69a80a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69c0103b70 0x7f69c0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f69c01031b0 0x7f69c0074340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 --2- 172.21.15.26:0/1454785740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69c0101fc0 0x7f69c0075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 >> 172.21.15.26:0/1454785740 conn(0x7f69c00fd550 msgr2=0x7f69c01064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 shutdown_connections 2024-04-23T18:15:02.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.766+0000 7f69c5736640 1 -- 172.21.15.26:0/1454785740 wait complete. 2024-04-23T18:15:02.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:02.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:15:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 -- 172.21.15.26:0/791810225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804100e60 msgr2=0x7fe804101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/791810225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804100e60 0x7fe804101280 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009900 tx=0x7fe7ec02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 -- 172.21.15.26:0/791810225 shutdown_connections 2024-04-23T18:15:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/791810225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804102a00 0x7fe804109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/791810225 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102040 0x7fe8041024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/791810225 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804100e60 0x7fe804101280 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 -- 172.21.15.26:0/791810225 >> 172.21.15.26:0/791810225 conn(0x7fe8040fc440 msgr2=0x7fe8040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 -- 172.21.15.26:0/791810225 shutdown_connections 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.887+0000 7fe80a5a2640 1 -- 172.21.15.26:0/791810225 wait complete. 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 Processor -- start 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 -- start start 2024-04-23T18:15:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804100e60 0x7fe804198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102a00 0x7fe80419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe80410ca20 con 0x7fe804102040 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe80410c8a0 con 0x7fe804100e60 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe80a5a2640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe80410cba0 con 0x7fe804102a00 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe8037fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.888+0000 7fe8037fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35108/0 (socket says 172.21.15.26:35108) 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 -- 172.21.15.26:0/4008897163 learned_addr learned my addr 172.21.15.26:0/4008897163 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe803fff640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804100e60 0x7fe804198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe808b3f640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102a00 0x7fe80419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 -- 172.21.15.26:0/4008897163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102a00 msgr2=0x7fe80419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102a00 0x7fe80419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 -- 172.21.15.26:0/4008897163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804100e60 msgr2=0x7fe804198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804100e60 0x7fe804198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 -- 172.21.15.26:0/4008897163 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80419fe60 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8037fe640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d950 tx=0x7fe7f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4017030 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8041a0150 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7f4004020 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40047a0 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.889+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8041a06f0 con 0x7fe804102040 2024-04-23T18:15:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.890+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8041a09b0 con 0x7fe804102040 2024-04-23T18:15:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.892+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe7f4009a30 con 0x7fe804102040 2024-04-23T18:15:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.893+0000 7fe8017fa640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 0x7fe7f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.894+0000 7fe803fff640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 0x7fe7f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.894+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe7f40cffe0 con 0x7fe804102040 2024-04-23T18:15:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.894+0000 7fe803fff640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 0x7fe7f00a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec005ab0 tx=0x7fe7ec005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:02.897+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7f409ce60 con 0x7fe804102040 2024-04-23T18:15:03.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.084+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8041a0b80 con 0x7fe804102040 2024-04-23T18:15:03.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.084+0000 7fe8017fa640 1 -- 172.21.15.26:0/4008897163 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fe7f409c760 con 0x7fe804102040 2024-04-23T18:15:03.078 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:15:03.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 msgr2=0x7fe7f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 0x7fe7f00a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec005ab0 tx=0x7fe7ec005a20 comp rx=0 tx=0).stop 2024-04-23T18:15:03.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 msgr2=0x7fe804198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d950 tx=0x7fe7f400de10 comp rx=0 tx=0).stop 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 shutdown_connections 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe7f00a3bb0 0x7fe7f00a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe804102a00 0x7fe80419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe804102040 0x7fe804198780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 --2- 172.21.15.26:0/4008897163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe804100e60 0x7fe804198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.086+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 >> 172.21.15.26:0/4008897163 conn(0x7fe8040fc440 msgr2=0x7fe80406a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.087+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 shutdown_connections 2024-04-23T18:15:03.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.087+0000 7fe80a5a2640 1 -- 172.21.15.26:0/4008897163 wait complete. 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2247773855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:03.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:03.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1454785740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:02 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T18:15:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2247773855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1454785740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:02 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- 172.21.15.26:0/1928125734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec101fe0 msgr2=0x7f42ec102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 --2- 172.21.15.26:0/1928125734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec101fe0 0x7f42ec102460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009900 tx=0x7f42e802a410 comp rx=0 tx=0).stop 2024-04-23T18:15:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- 172.21.15.26:0/1928125734 shutdown_connections 2024-04-23T18:15:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 --2- 172.21.15.26:0/1928125734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec101fe0 0x7f42ec102460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 --2- 172.21.15.26:0/1928125734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec1029a0 0x7f42ec1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 --2- 172.21.15.26:0/1928125734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 0x7f42ec1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- 172.21.15.26:0/1928125734 >> 172.21.15.26:0/1928125734 conn(0x7f42ec0fc400 msgr2=0x7f42ec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- 172.21.15.26:0/1928125734 shutdown_connections 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- 172.21.15.26:0/1928125734 wait complete. 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 Processor -- start 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.203+0000 7f42f342f640 1 -- start start 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 0x7f42ec19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec1029a0 0x7f42ec1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10c9d0 con 0x7f42ec101fe0 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10c850 con 0x7f42ec1029a0 2024-04-23T18:15:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10cb50 con 0x7f42ec100e90 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35124/0 (socket says 172.21.15.26:35124) 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 -- 172.21.15.26:0/1966132333 learned_addr learned my addr 172.21.15.26:0/1966132333 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f11cb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 0x7f42ec19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f19cc640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec1029a0 0x7f42ec1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 -- 172.21.15.26:0/1966132333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 msgr2=0x7f42ec19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 0x7f42ec19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 -- 172.21.15.26:0/1966132333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec1029a0 msgr2=0x7f42ec1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec1029a0 0x7f42ec1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 -- 172.21.15.26:0/1966132333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec1a2020 con 0x7f42ec101fe0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f09ca640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006010 tx=0x7f42e80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e803f030 con 0x7f42ec101fe0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e8031910 con 0x7f42ec101fe0 2024-04-23T18:15:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ec1a22b0 con 0x7f42ec101fe0 2024-04-23T18:15:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8035070 con 0x7f42ec101fe0 2024-04-23T18:15:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.204+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ec1a27a0 con 0x7f42ec101fe0 2024-04-23T18:15:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.206+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42ec1a2a00 con 0x7f42ec101fe0 2024-04-23T18:15:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.208+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f42e8042030 con 0x7f42ec101fe0 2024-04-23T18:15:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.208+0000 7f42e27fc640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.208+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f42e80f0840 con 0x7f42ec101fe0 2024-04-23T18:15:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.208+0000 7f42f11cb640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.209+0000 7f42f11cb640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 0x7f42d80a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009bd0 tx=0x7f42dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.211+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42e80bd070 con 0x7f42ec101fe0 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2247773855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:03.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:03.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:03.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1454785740' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.243 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:02 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:03.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.410+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f42ec1a30f0 con 0x7f42ec101fe0 2024-04-23T18:15:03.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.411+0000 7f42e27fc640 1 -- 172.21.15.26:0/1966132333 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f42e80c1370 con 0x7f42ec101fe0 2024-04-23T18:15:03.404 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout:[ 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: "poolnum": 1, 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: "poolname": ".mgr" 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:15:03.405 INFO:teuthology.orchestra.run.smithi026.stdout:] 2024-04-23T18:15:03.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.412+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 msgr2=0x7f42d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.412+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 0x7f42d80a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009bd0 tx=0x7f42dc005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.412+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 msgr2=0x7f42ec19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.412+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006010 tx=0x7f42e80042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 shutdown_connections 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42ec1029a0 0x7f42ec1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42ec101fe0 0x7f42ec19a940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 --2- 172.21.15.26:0/1966132333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42ec100e90 0x7f42ec19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 >> 172.21.15.26:0/1966132333 conn(0x7f42ec0fc400 msgr2=0x7f42ec0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 shutdown_connections 2024-04-23T18:15:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.413+0000 7f42f342f640 1 -- 172.21.15.26:0/1966132333 wait complete. 2024-04-23T18:15:03.419 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T18:15:03.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T18:15:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 -- 172.21.15.26:0/4024965339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4100280 msgr2=0x7fb9c41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/4024965339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4100280 0x7fb9c41006a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009cd0 tx=0x7fb9b402a8e0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 -- 172.21.15.26:0/4024965339 shutdown_connections 2024-04-23T18:15:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/4024965339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c4103cb0 0x7fb9c410a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/4024965339 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c41032f0 0x7fb9c4103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/4024965339 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4100280 0x7fb9c41006a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 -- 172.21.15.26:0/4024965339 >> 172.21.15.26:0/4024965339 conn(0x7fb9c40fbb90 msgr2=0x7fb9c40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 -- 172.21.15.26:0/4024965339 shutdown_connections 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.533+0000 7fb9c9a20640 1 -- 172.21.15.26:0/4024965339 wait complete. 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 Processor -- start 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 -- start start 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c4100280 0x7fb9c41982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c41032f0 0x7fb9c4198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410daf0 con 0x7fb9c4103cb0 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410d970 con 0x7fb9c41032f0 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c9a20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410dc70 con 0x7fb9c4100280 2024-04-23T18:15:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35146/0 (socket says 172.21.15.26:35146) 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 -- 172.21.15.26:0/521587102 learned_addr learned my addr 172.21.15.26:0/521587102 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c2ffd640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c4100280 0x7fb9c41982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c27fc640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c41032f0 0x7fb9c4198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 -- 172.21.15.26:0/521587102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c4100280 msgr2=0x7fb9c41982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c4100280 0x7fb9c41982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 -- 172.21.15.26:0/521587102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c41032f0 msgr2=0x7fb9c4198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.534+0000 7fb9c37fe640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c41032f0 0x7fb9c4198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c37fe640 1 -- 172.21.15.26:0/521587102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c419fee0 con 0x7fb9c4103cb0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c37fe640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c970 tx=0x7fb9bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc007bb0 con 0x7fb9c4103cb0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9bc007d50 con 0x7fb9c4103cb0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c41a01d0 con 0x7fb9c4103cb0 2024-04-23T18:15:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc004e60 con 0x7fb9c4103cb0 2024-04-23T18:15:03.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.535+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c41a06d0 con 0x7fb9c4103cb0 2024-04-23T18:15:03.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.536+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c4109560 con 0x7fb9c4103cb0 2024-04-23T18:15:03.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.537+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb9bc005020 con 0x7fb9c4103cb0 2024-04-23T18:15:03.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.538+0000 7fb9c8a1e640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 0x7fb9ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:03.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.538+0000 7fb9c2ffd640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 0x7fb9ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:03.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.538+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb9bc0cfc90 con 0x7fb9c4103cb0 2024-04-23T18:15:03.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.538+0000 7fb9c2ffd640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 0x7fb9ac0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009ca0 tx=0x7fb9b4005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.542+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9bc014030 con 0x7fb9c4103cb0 2024-04-23T18:15:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:03.731+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb9c4062b50 con 0x7fb9c4103cb0 2024-04-23T18:15:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:03 smithi110 ceph-mon[32585]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4008897163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1966132333' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:15:03.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:15:04.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:03 smithi026 ceph-mon[29364]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:04.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4008897163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:04.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1966132333' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:15:04.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:15:04.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:03 smithi131 ceph-mon[33041]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:04.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4008897163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:04.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1966132333' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:15:04.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:15:04.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.665+0000 7fb9c8a1e640 1 -- 172.21.15.26:0/521587102 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fb9bc09ca70 con 0x7fb9c4103cb0 2024-04-23T18:15:04.659 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' created 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 msgr2=0x7fb9ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 0x7fb9ac0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009ca0 tx=0x7fb9b4005f70 comp rx=0 tx=0).stop 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 msgr2=0x7fb9c419f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c970 tx=0x7fb9bc00ce30 comp rx=0 tx=0).stop 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 shutdown_connections 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9ac0a3bb0 0x7fb9ac0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c41032f0 0x7fb9c4198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9c4103cb0 0x7fb9c419f870 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 --2- 172.21.15.26:0/521587102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9c4100280 0x7fb9c41982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 >> 172.21.15.26:0/521587102 conn(0x7fb9c40fbb90 msgr2=0x7fb9c4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:04.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 shutdown_connections 2024-04-23T18:15:04.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.668+0000 7fb9c9a20640 1 -- 172.21.15.26:0/521587102 wait complete. 2024-04-23T18:15:04.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T18:15:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 -- 172.21.15.26:0/2418855736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c101fa0 msgr2=0x7f2e6c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 --2- 172.21.15.26:0/2418855736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1023c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009900 tx=0x7f2e6802a410 comp rx=0 tx=0).stop 2024-04-23T18:15:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 -- 172.21.15.26:0/2418855736 shutdown_connections 2024-04-23T18:15:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 --2- 172.21.15.26:0/2418855736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 0x7f2e6c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 --2- 172.21.15.26:0/2418855736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c103190 0x7f2e6c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 --2- 172.21.15.26:0/2418855736 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1023c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 -- 172.21.15.26:0/2418855736 >> 172.21.15.26:0/2418855736 conn(0x7f2e6c0fd530 msgr2=0x7f2e6c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 -- 172.21.15.26:0/2418855736 shutdown_connections 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.791+0000 7f2e745e1640 1 -- 172.21.15.26:0/2418855736 wait complete. 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 Processor -- start 2024-04-23T18:15:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 -- start start 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 0x7f2e6c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c10dae0 con 0x7f2e6c103190 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c10d960 con 0x7f2e6c103b50 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e745e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c10dc60 con 0x7f2e6c101fa0 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e71b7c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e71b7c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35166/0 (socket says 172.21.15.26:35166) 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e71b7c640 1 -- 172.21.15.26:0/354570437 learned_addr learned my addr 172.21.15.26:0/354570437 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e72b7e640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 0x7f2e6c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e7237d640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e71b7c640 1 -- 172.21.15.26:0/354570437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c101fa0 msgr2=0x7f2e6c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.792+0000 7f2e71b7c640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e71b7c640 1 -- 172.21.15.26:0/354570437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 msgr2=0x7f2e6c19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e71b7c640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 0x7f2e6c19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e71b7c640 1 -- 172.21.15.26:0/354570437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c19fe10 con 0x7f2e6c103190 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e71b7c640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00d950 tx=0x7f2e5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c017030 con 0x7f2e6c103190 2024-04-23T18:15:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c004020 con 0x7f2e6c103190 2024-04-23T18:15:04.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c1a0100 con 0x7f2e6c103190 2024-04-23T18:15:04.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0047a0 con 0x7f2e6c103190 2024-04-23T18:15:04.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.793+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c1a06a0 con 0x7f2e6c103190 2024-04-23T18:15:04.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.794+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6c109380 con 0x7f2e6c103190 2024-04-23T18:15:04.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.795+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2e5c009a30 con 0x7f2e6c103190 2024-04-23T18:15:04.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.796+0000 7f2e637fe640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 0x7f2e580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:04.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.796+0000 7f2e7237d640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 0x7f2e580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:04.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.796+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f2e5c0d0220 con 0x7f2e6c103190 2024-04-23T18:15:04.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.796+0000 7f2e7237d640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 0x7f2e580a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2e68000c00 tx=0x7f2e680314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:04.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.800+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e5c09cfb0 con 0x7f2e6c103190 2024-04-23T18:15:04.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:04.986+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2e6c02d5c0 con 0x7f2e6c103190 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.671+0000 7f2e637fe640 1 -- 172.21.15.26:0/354570437 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f2e5c09c8b0 con 0x7f2e6c103190 2024-04-23T18:15:05.665 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' created 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 msgr2=0x7f2e580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 0x7f2e580a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2e68000c00 tx=0x7f2e680314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 msgr2=0x7f2e6c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00d950 tx=0x7f2e5c00de10 comp rx=0 tx=0).stop 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 shutdown_connections 2024-04-23T18:15:05.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e580a3c30 0x7f2e580a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e6c103b50 0x7f2e6c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e6c103190 0x7f2e6c198730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 --2- 172.21.15.26:0/354570437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c101fa0 0x7f2e6c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.676+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 >> 172.21.15.26:0/354570437 conn(0x7f2e6c0fd530 msgr2=0x7f2e6c106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:05.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.678+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 shutdown_connections 2024-04-23T18:15:05.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.679+0000 7f2e745e1640 1 -- 172.21.15.26:0/354570437 wait complete. 2024-04-23T18:15:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:05.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:05 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:05.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:15:05.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T18:15:05.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/521587102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:05.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:05 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/3451929406 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 msgr2=0x7f4ecc107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/3451929406 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc107750 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009900 tx=0x7f4ebc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/3451929406 shutdown_connections 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/3451929406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/3451929406 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc107750 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/3451929406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ecc102a30 0x7f4ecc104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/3451929406 >> 172.21.15.26:0/3451929406 conn(0x7f4ecc0fc440 msgr2=0x7f4ecc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/3451929406 shutdown_connections 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.812+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/3451929406 wait complete. 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 Processor -- start 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 -- start start 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ecc102a30 0x7f4ecc198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc10dab0 con 0x7f4ecc105360 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc10d930 con 0x7f4ecc107c90 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ed0b6f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc10dc30 con 0x7f4ecc102a30 2024-04-23T18:15:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35182/0 (socket says 172.21.15.26:35182) 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 -- 172.21.15.26:0/4279330128 learned_addr learned my addr 172.21.15.26:0/4279330128 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ecad9d640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 -- 172.21.15.26:0/4279330128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ecc102a30 msgr2=0x7f4ecc198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ecc102a30 0x7f4ecc198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 -- 172.21.15.26:0/4279330128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 msgr2=0x7f4ecc19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ec9d9b640 1 -- 172.21.15.26:0/4279330128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ecc19fe90 con 0x7f4ecc105360 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.813+0000 7f4ecad9d640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ec9d9b640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc006010 tx=0x7f4ebc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc03f030 con 0x7f4ecc105360 2024-04-23T18:15:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ebc003cb0 con 0x7f4ecc105360 2024-04-23T18:15:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ecc1a0120 con 0x7f4ecc105360 2024-04-23T18:15:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc035070 con 0x7f4ecc105360 2024-04-23T18:15:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.814+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ecc1a0660 con 0x7f4ecc105360 2024-04-23T18:15:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.815+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ecc0fdf30 con 0x7f4ecc105360 2024-04-23T18:15:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.816+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4ebc042030 con 0x7f4ecc105360 2024-04-23T18:15:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.817+0000 7f4ebb7fe640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 0x7f4eb40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.817+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f4ebc0f0cd0 con 0x7f4ecc105360 2024-04-23T18:15:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.817+0000 7f4eca59c640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 0x7f4eb40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.817+0000 7f4eca59c640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 0x7f4eb40a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000ba80 tx=0x7f4ec0005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:05.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:05.820+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ebc0bd860 con 0x7f4ecc105360 2024-04-23T18:15:06.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.047+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4ecc02d5c0 con 0x7f4ecc105360 2024-04-23T18:15:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.672+0000 7f4ebb7fe640 1 -- 172.21.15.26:0/4279330128 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f4ebc0bd160 con 0x7f4ecc105360 2024-04-23T18:15:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 msgr2=0x7f4eb40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 0x7f4eb40a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000ba80 tx=0x7f4ec0005ed0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 msgr2=0x7f4ecc1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc006010 tx=0x7f4ebc0042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 shutdown_connections 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4eb40a3c30 0x7f4eb40a60f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ecc107c90 0x7f4ecc19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4ecc105360 0x7f4ecc1987b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 --2- 172.21.15.26:0/4279330128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ecc102a30 0x7f4ecc198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 >> 172.21.15.26:0/4279330128 conn(0x7f4ecc0fc440 msgr2=0x7f4ecc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 shutdown_connections 2024-04-23T18:15:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.675+0000 7f4ed0b6f640 1 -- 172.21.15.26:0/4279330128 wait complete. 2024-04-23T18:15:06.684 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:06.684 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:15:06.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:15:06.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:15:06.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:15:06.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:06 smithi026 ceph-mon[29364]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:15:06.686 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:15:06.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:06 smithi110 ceph-mon[32585]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:15:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:15:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/354570437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:15:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:15:06.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:15:06.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:06 smithi131 ceph-mon[33041]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:15:06.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- 172.21.15.26:0/3175471159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334100e90 msgr2=0x7f83341012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 --2- 172.21.15.26:0/3175471159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334100e90 0x7f83341012b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8330009900 tx=0x7f833002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- 172.21.15.26:0/3175471159 shutdown_connections 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 --2- 172.21.15.26:0/3175471159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f83341092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 --2- 172.21.15.26:0/3175471159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334101fe0 0x7f8334102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 --2- 172.21.15.26:0/3175471159 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334100e90 0x7f83341012b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- 172.21.15.26:0/3175471159 >> 172.21.15.26:0/3175471159 conn(0x7f83340fc400 msgr2=0x7f83340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- 172.21.15.26:0/3175471159 shutdown_connections 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- 172.21.15.26:0/3175471159 wait complete. 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 Processor -- start 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.801+0000 7f833be43640 1 -- start start 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334100e90 0x7f8334198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f833419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8334069d90 con 0x7f8334101fe0 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8334069c10 con 0x7f83341029a0 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8334069f10 con 0x7f8334100e90 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35200/0 (socket says 172.21.15.26:35200) 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 -- 172.21.15.26:0/1944701161 learned_addr learned my addr 172.21.15.26:0/1944701161 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f8339bdf640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334100e90 0x7f8334198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833a3e0640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f833419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 -- 172.21.15.26:0/1944701161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334100e90 msgr2=0x7f8334198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334100e90 0x7f8334198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 -- 172.21.15.26:0/1944701161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 msgr2=0x7f833419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f833419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 -- 172.21.15.26:0/1944701161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833419fe40 con 0x7f8334101fe0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833a3e0640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f833419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f83393de640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f832400b450 tx=0x7f832400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324015020 con 0x7f8334101fe0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83341a0130 con 0x7f8334101fe0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8324011d40 con 0x7f8334101fe0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.802+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832400f340 con 0x7f8334101fe0 2024-04-23T18:15:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.803+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83341a06b0 con 0x7f8334101fe0 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.804+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83341081a0 con 0x7f8334101fe0 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.805+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8324018020 con 0x7f8334101fe0 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.805+0000 7f832affd640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 0x7f83200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.805+0000 7f8339bdf640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 0x7f83200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.805+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f83240cfdb0 con 0x7f8334101fe0 2024-04-23T18:15:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.806+0000 7f8339bdf640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 0x7f83200a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8330000c00 tx=0x7f83300314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:06.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.810+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f832401f020 con 0x7f8334101fe0 2024-04-23T18:15:06.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:06.998+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f833402d810 con 0x7f8334101fe0 2024-04-23T18:15:06.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.000+0000 7f832affd640 1 -- 172.21.15.26:0/1944701161 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22244 (secure 0 0 0) 0x7f83240cde20 con 0x7f8334101fe0 2024-04-23T18:15:06.994 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:06.994 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":85,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:15:06.048398+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:06.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 msgr2=0x7f83200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 0x7f83200a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8330000c00 tx=0x7f83300314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 msgr2=0x7f8334198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f832400b450 tx=0x7f832400b910 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 shutdown_connections 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f83200a3bb0 0x7f83200a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83341029a0 0x7f833419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8334101fe0 0x7f8334198760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 --2- 172.21.15.26:0/1944701161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8334100e90 0x7f8334198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 >> 172.21.15.26:0/1944701161 conn(0x7f83340fc400 msgr2=0x7f8334105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 shutdown_connections 2024-04-23T18:15:06.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.002+0000 7f833be43640 1 -- 172.21.15.26:0/1944701161 wait complete. 2024-04-23T18:15:07.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:07.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 -- 172.21.15.26:0/1476055115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c100e60 msgr2=0x7f313c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 --2- 172.21.15.26:0/1476055115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c100e60 0x7f313c101280 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f312c009900 tx=0x7f312c02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 -- 172.21.15.26:0/1476055115 shutdown_connections 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 --2- 172.21.15.26:0/1476055115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102040 0x7f313c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 --2- 172.21.15.26:0/1476055115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c102a00 0x7f313c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 --2- 172.21.15.26:0/1476055115 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c100e60 0x7f313c101280 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 -- 172.21.15.26:0/1476055115 >> 172.21.15.26:0/1476055115 conn(0x7f313c0fc440 msgr2=0x7f313c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 -- 172.21.15.26:0/1476055115 shutdown_connections 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.115+0000 7f31424dc640 1 -- 172.21.15.26:0/1476055115 wait complete. 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 Processor -- start 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 -- start start 2024-04-23T18:15:07.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c100e60 0x7f313c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 0x7f313c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f313c10ca60 con 0x7f313c102040 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f313c10c8e0 con 0x7f313c102a00 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f31424dc640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f313c10cbe0 con 0x7f313c100e60 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f313b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f313b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35220/0 (socket says 172.21.15.26:35220) 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f313b7fe640 1 -- 172.21.15.26:0/410488495 learned_addr learned my addr 172.21.15.26:0/410488495 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.116+0000 7f3140a79640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 0x7f313c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313bfff640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c100e60 0x7f313c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 -- 172.21.15.26:0/410488495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c100e60 msgr2=0x7f313c198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c100e60 0x7f313c198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 -- 172.21.15.26:0/410488495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 msgr2=0x7f313c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 0x7f313c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 -- 172.21.15.26:0/410488495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313c19fe30 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f3140a79640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 0x7f313c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f313b7fe640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f313000c920 tx=0x7f313000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3130007c70 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3130007e10 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313c1a0120 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3130018820 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.117+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f313c1a0740 con 0x7f313c102040 2024-04-23T18:15:07.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.118+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f313c1a0a00 con 0x7f313c102040 2024-04-23T18:15:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.119+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f313000f450 con 0x7f313c102040 2024-04-23T18:15:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.120+0000 7f31397fa640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 0x7f31240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.120+0000 7f313bfff640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 0x7f31240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.120+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f31300d06e0 con 0x7f313c102040 2024-04-23T18:15:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.120+0000 7f313bfff640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 0x7f31240a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:07.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.123+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3130014030 con 0x7f313c102040 2024-04-23T18:15:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.312+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f313c1a0bd0 con 0x7f313c102040 2024-04-23T18:15:07.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.313+0000 7f31397fa640 1 -- 172.21.15.26:0/410488495 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22244 (secure 0 0 0) 0x7f313009d320 con 0x7f313c102040 2024-04-23T18:15:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:07.307 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":85,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:15:06.048398+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 msgr2=0x7f31240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 0x7f31240a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c0314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 msgr2=0x7f313c1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f313000c920 tx=0x7f313000cde0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 shutdown_connections 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f31240a3c30 0x7f31240a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f313c102a00 0x7f313c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f313c102040 0x7f313c1987c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 --2- 172.21.15.26:0/410488495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f313c100e60 0x7f313c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 >> 172.21.15.26:0/410488495 conn(0x7f313c0fc440 msgr2=0x7f313c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:07.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 shutdown_connections 2024-04-23T18:15:07.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.315+0000 7f31424dc640 1 -- 172.21.15.26:0/410488495 wait complete. 2024-04-23T18:15:07.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 -- 172.21.15.26:0/1377849470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c10107cd0 msgr2=0x7f3c1010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 --2- 172.21.15.26:0/1377849470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c10107cd0 0x7f3c1010a0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009ca0 tx=0x7f3c0802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 -- 172.21.15.26:0/1377849470 shutdown_connections 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 --2- 172.21.15.26:0/1377849470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c101053a0 0x7f3c10107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 --2- 172.21.15.26:0/1377849470 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c10107cd0 0x7f3c1010a0c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 --2- 172.21.15.26:0/1377849470 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 0x7f3c10104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 -- 172.21.15.26:0/1377849470 >> 172.21.15.26:0/1377849470 conn(0x7f3c100fc400 msgr2=0x7f3c100fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 -- 172.21.15.26:0/1377849470 shutdown_connections 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.435+0000 7f3c17975640 1 -- 172.21.15.26:0/1377849470 wait complete. 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 Processor -- start 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 -- start start 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 0x7f3c10198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10107cd0 0x7f3c1019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1010db30 con 0x7f3c101053a0 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1010d9b0 con 0x7f3c10107cd0 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c17975640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1010dcb0 con 0x7f3c10102a70 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35226/0 (socket says 172.21.15.26:35226) 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 -- 172.21.15.26:0/1917137218 learned_addr learned my addr 172.21.15.26:0/1917137218 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c15711640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 0x7f3c10198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 -- 172.21.15.26:0/1917137218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 msgr2=0x7f3c10198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 0x7f3c10198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 -- 172.21.15.26:0/1917137218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10107cd0 msgr2=0x7f3c1019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10107cd0 0x7f3c1019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 -- 172.21.15.26:0/1917137218 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1019fe50 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.436+0000 7f3c14f10640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000e9c0 tx=0x7f3c0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.437+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000cd80 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.437+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c000054d0 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.437+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c101a0140 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.437+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00010430 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.437+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c101a06e0 con 0x7f3c101053a0 2024-04-23T18:15:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.438+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c100fdef0 con 0x7f3c101053a0 2024-04-23T18:15:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.439+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3c00004fd0 con 0x7f3c101053a0 2024-04-23T18:15:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.439+0000 7f3c067fc640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 0x7f3bfc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.439+0000 7f3c15711640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 0x7f3bfc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.439+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3c000d0760 con 0x7f3c101053a0 2024-04-23T18:15:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.440+0000 7f3c15711640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 0x7f3bfc0a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009b60 tx=0x7f3c0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.444+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c0009d290 con 0x7f3c101053a0 2024-04-23T18:15:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:07.675+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3c1002d7c0 con 0x7f3c101053a0 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1944701161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:07 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/410488495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:07.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:15:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:15:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:15:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:15:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1944701161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:07 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/410488495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4279330128' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1944701161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:07.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:07 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/410488495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:08.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.679+0000 7f3c067fc640 1 -- 172.21.15.26:0/1917137218 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f3c0009cb90 con 0x7f3c101053a0 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 msgr2=0x7f3bfc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 0x7f3bfc0a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009b60 tx=0x7f3c0c005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 msgr2=0x7f3c10198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000e9c0 tx=0x7f3c0000ee80 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 shutdown_connections 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f3bfc0a3c30 0x7f3bfc0a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10107cd0 0x7f3c1019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c101053a0 0x7f3c10198770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 --2- 172.21.15.26:0/1917137218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c10102a70 0x7f3c10198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 >> 172.21.15.26:0/1917137218 conn(0x7f3c100fc400 msgr2=0x7f3c10103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 shutdown_connections 2024-04-23T18:15:08.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.683+0000 7f3c17975640 1 -- 172.21.15.26:0/1917137218 wait complete. 2024-04-23T18:15:08.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: pgmap v283: 97 pgs: 2 creating+activating, 26 creating+peering, 19 active+clean, 50 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:08.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:15:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:15:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: Cluster is now healthy 2024-04-23T18:15:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:15:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:active 2024-04-23T18:15:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:08 smithi026 ceph-mon[29364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:08.691 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:15:08.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-23T18:15:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: pgmap v283: 97 pgs: 2 creating+activating, 26 creating+peering, 19 active+clean, 50 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:15:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:15:08.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: Cluster is now healthy 2024-04-23T18:15:08.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:15:08.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:active 2024-04-23T18:15:08.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:08 smithi110 ceph-mon[32585]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: pgmap v283: 97 pgs: 2 creating+activating, 26 creating+peering, 19 active+clean, 50 unknown; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: Cluster is now healthy 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] up:active 2024-04-23T18:15:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:08 smithi131 ceph-mon[33041]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 -- 172.21.15.26:0/3914547931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c100e60 msgr2=0x7f415c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 --2- 172.21.15.26:0/3914547931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c100e60 0x7f415c101280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4150009900 tx=0x7f415002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 -- 172.21.15.26:0/3914547931 shutdown_connections 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 --2- 172.21.15.26:0/3914547931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c102040 0x7f415c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 --2- 172.21.15.26:0/3914547931 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 --2- 172.21.15.26:0/3914547931 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c100e60 0x7f415c101280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 -- 172.21.15.26:0/3914547931 >> 172.21.15.26:0/3914547931 conn(0x7f415c0fc440 msgr2=0x7f415c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 -- 172.21.15.26:0/3914547931 shutdown_connections 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 -- 172.21.15.26:0/3914547931 wait complete. 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.809+0000 7f41631f0640 1 Processor -- start 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 -- start start 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c100e60 0x7f415c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10db90 con 0x7f415c102040 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10da10 con 0x7f415c100e60 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f41631f0640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10dd10 con 0x7f415c102a00 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38022/0 (socket says 172.21.15.26:38022) 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 -- 172.21.15.26:0/1686470173 learned_addr learned my addr 172.21.15.26:0/1686470173 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f4160f8c640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c100e60 0x7f415c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f416178d640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 -- 172.21.15.26:0/1686470173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 msgr2=0x7f415c19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 -- 172.21.15.26:0/1686470173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c100e60 msgr2=0x7f415c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c100e60 0x7f415c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 -- 172.21.15.26:0/1686470173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c19fdc0 con 0x7f415c102040 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f416178d640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.810+0000 7f415bfff640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f414800c920 tx=0x7f414800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.811+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148007c70 con 0x7f415c102040 2024-04-23T18:15:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.811+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4148007e10 con 0x7f415c102040 2024-04-23T18:15:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.811+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c1a00b0 con 0x7f415c102040 2024-04-23T18:15:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.811+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148018820 con 0x7f415c102040 2024-04-23T18:15:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.811+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c1a0600 con 0x7f415c102040 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.812+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415c108280 con 0x7f415c102040 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.813+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f414800f450 con 0x7f415c102040 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.813+0000 7f4159ffb640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 0x7f414c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.813+0000 7f4160f8c640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 0x7f414c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.813+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f41480d03c0 con 0x7f415c102040 2024-04-23T18:15:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.814+0000 7f4160f8c640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 0x7f414c0a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4150005ab0 tx=0x7f4150005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:08.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:08.818+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4148014030 con 0x7f415c102040 2024-04-23T18:15:09.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.044+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f415c02d5c0 con 0x7f415c102040 2024-04-23T18:15:09.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.684+0000 7f4159ffb640 1 -- 172.21.15.26:0/1686470173 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7f414809cef0 con 0x7f415c102040 2024-04-23T18:15:09.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 msgr2=0x7f414c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 0x7f414c0a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4150005ab0 tx=0x7f4150005a20 comp rx=0 tx=0).stop 2024-04-23T18:15:09.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 msgr2=0x7f415c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f414800c920 tx=0x7f414800cde0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 shutdown_connections 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f414c0a3bb0 0x7f414c0a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f415c102a00 0x7f415c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f415c102040 0x7f415c1986e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 --2- 172.21.15.26:0/1686470173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415c100e60 0x7f415c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.687+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 >> 172.21.15.26:0/1686470173 conn(0x7f415c0fc440 msgr2=0x7f415c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.688+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 shutdown_connections 2024-04-23T18:15:09.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.688+0000 7f41631f0640 1 -- 172.21.15.26:0/1686470173 wait complete. 2024-04-23T18:15:09.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-23T18:15:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 -- 172.21.15.26:0/1219760027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc1053a0 msgr2=0x7f78bc107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 --2- 172.21.15.26:0/1219760027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc1053a0 0x7f78bc107790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009900 tx=0x7f78a402a410 comp rx=0 tx=0).stop 2024-04-23T18:15:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 -- 172.21.15.26:0/1219760027 shutdown_connections 2024-04-23T18:15:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 --2- 172.21.15.26:0/1219760027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc107cd0 0x7f78bc10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 --2- 172.21.15.26:0/1219760027 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc1053a0 0x7f78bc107790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 --2- 172.21.15.26:0/1219760027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 0x7f78bc104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 -- 172.21.15.26:0/1219760027 >> 172.21.15.26:0/1219760027 conn(0x7f78bc0fc400 msgr2=0x7f78bc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 -- 172.21.15.26:0/1219760027 shutdown_connections 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.816+0000 7f78c1736640 1 -- 172.21.15.26:0/1219760027 wait complete. 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 Processor -- start 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 -- start start 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 0x7f78bc1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc1053a0 0x7f78bc100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc10dac0 con 0x7f78bc107cd0 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc10d940 con 0x7f78bc102a70 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78c1736640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc10dc40 con 0x7f78bc1053a0 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78bb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78bb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38036/0 (socket says 172.21.15.26:38036) 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78bb7fe640 1 -- 172.21.15.26:0/2961569454 learned_addr learned my addr 172.21.15.26:0/2961569454 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78ba7fc640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc1053a0 0x7f78bc100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.817+0000 7f78baffd640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 0x7f78bc1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 -- 172.21.15.26:0/2961569454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc1053a0 msgr2=0x7f78bc100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc1053a0 0x7f78bc100a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 -- 172.21.15.26:0/2961569454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 msgr2=0x7f78bc1023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 0x7f78bc1023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 -- 172.21.15.26:0/2961569454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc1017f0 con 0x7f78bc107cd0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78bb7fe640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00cad0 tx=0x7f78ac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00ae20 con 0x7f78bc107cd0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78ac005020 con 0x7f78bc107cd0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78bc1a23a0 con 0x7f78bc107cd0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00f6b0 con 0x7f78bc107cd0 2024-04-23T18:15:09.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.818+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78bc1a28c0 con 0x7f78bc107cd0 2024-04-23T18:15:09.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.819+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78bc068d70 con 0x7f78bc107cd0 2024-04-23T18:15:09.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.820+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f78ac0051d0 con 0x7f78bc107cd0 2024-04-23T18:15:09.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.821+0000 7f78b3fff640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 0x7f78a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:09.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.821+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f78ac0d0280 con 0x7f78bc107cd0 2024-04-23T18:15:09.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.821+0000 7f78baffd640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 0x7f78a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:09.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.821+0000 7f78baffd640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 0x7f78a80a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f78b40058c0 tx=0x7f78b400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:09.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:09.825+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78ac014030 con 0x7f78bc107cd0 2024-04-23T18:15:09.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:09 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:15:09.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:09 smithi110 ceph-mon[32585]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:09.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:09 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:15:09.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:09 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:15:09.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:09 smithi026 ceph-mon[29364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:09.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:09 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:15:09.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:09 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1917137218' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:15:09.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:09 smithi131 ceph-mon[33041]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:09.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:09 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:15:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.052+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f78bc062b50 con 0x7f78bc107cd0 2024-04-23T18:15:10.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.693+0000 7f78b3fff640 1 -- 172.21.15.26:0/2961569454 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f78ac09c760 con 0x7f78bc107cd0 2024-04-23T18:15:10.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.695+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 msgr2=0x7f78a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.695+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 0x7f78a80a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f78b40058c0 tx=0x7f78b400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 msgr2=0x7f78bc100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00cad0 tx=0x7f78ac00cf90 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 shutdown_connections 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78a80a3bb0 0x7f78a80a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f78bc107cd0 0x7f78bc100f60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78bc1053a0 0x7f78bc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 --2- 172.21.15.26:0/2961569454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78bc102a70 0x7f78bc1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 >> 172.21.15.26:0/2961569454 conn(0x7f78bc0fc400 msgr2=0x7f78bc1034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 shutdown_connections 2024-04-23T18:15:10.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.696+0000 7f78c1736640 1 -- 172.21.15.26:0/2961569454 wait complete. 2024-04-23T18:15:10.705 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T18:15:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 -- 172.21.15.26:0/481680943 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c105360 msgr2=0x7f570c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 --2- 172.21.15.26:0/481680943 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c105360 0x7f570c107750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5700009900 tx=0x7f570002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 -- 172.21.15.26:0/481680943 shutdown_connections 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 --2- 172.21.15.26:0/481680943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 0x7f570c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 --2- 172.21.15.26:0/481680943 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c105360 0x7f570c107750 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 --2- 172.21.15.26:0/481680943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c102a30 0x7f570c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 -- 172.21.15.26:0/481680943 >> 172.21.15.26:0/481680943 conn(0x7f570c0fc440 msgr2=0x7f570c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 -- 172.21.15.26:0/481680943 shutdown_connections 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.819+0000 7f5710e7b640 1 -- 172.21.15.26:0/481680943 wait complete. 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 Processor -- start 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 -- start start 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c105360 0x7f570c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 0x7f570c1a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f570c10daf0 con 0x7f570c102a30 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f570c10d970 con 0x7f570c107c90 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f5710e7b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f570c10dc70 con 0x7f570c105360 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f570a59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f570a59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38056/0 (socket says 172.21.15.26:38056) 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.820+0000 7f570a59c640 1 -- 172.21.15.26:0/757408528 learned_addr learned my addr 172.21.15.26:0/757408528 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570ad9d640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 0x7f570c1a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f5709d9b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c105360 0x7f570c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 -- 172.21.15.26:0/757408528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c105360 msgr2=0x7f570c19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c105360 0x7f570c19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 -- 172.21.15.26:0/757408528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 msgr2=0x7f570c1a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 0x7f570c1a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 -- 172.21.15.26:0/757408528 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c1a2090 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f570a59c640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c890 tx=0x7f56fc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc004220 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c1a2380 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56fc0043c0 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc004a00 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.821+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c1a28a0 con 0x7f570c102a30 2024-04-23T18:15:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.822+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f570c1097e0 con 0x7f570c102a30 2024-04-23T18:15:10.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.823+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f56fc0079b0 con 0x7f570c102a30 2024-04-23T18:15:10.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.824+0000 7f56fb7fe640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 0x7f56f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:10.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.824+0000 7f5709d9b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 0x7f56f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:10.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.824+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f56fc0d0090 con 0x7f570c102a30 2024-04-23T18:15:10.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.825+0000 7f5709d9b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 0x7f56f40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5700000c00 tx=0x7f57000314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:10.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:10.828+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56fc014030 con 0x7f570c102a30 2024-04-23T18:15:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: pgmap v285: 97 pgs: 2 creating+activating, 26 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 490 B/s wr, 1 op/s 2024-04-23T18:15:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:15:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:15:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:15:10.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:10.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:15:10.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:15:10.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:10 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: pgmap v285: 97 pgs: 2 creating+activating, 26 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 490 B/s wr, 1 op/s 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:15:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:10 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:15:10.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: pgmap v285: 97 pgs: 2 creating+activating, 26 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 490 B/s wr, 1 op/s 2024-04-23T18:15:10.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:15:10.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1686470173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:15:10.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:15:10.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:15:10.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T18:15:10.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T18:15:10.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:10 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:15:11.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.056+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f570c02d5c0 con 0x7f570c102a30 2024-04-23T18:15:11.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.650+0000 7f56fb7fe640 1 -- 172.21.15.26:0/757408528 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f56fc09cb20 con 0x7f570c102a30 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 msgr2=0x7f56f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 0x7f56f40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5700000c00 tx=0x7f57000314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 msgr2=0x7f570c19a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00c890 tx=0x7f56fc00cd50 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 shutdown_connections 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f56f40a3bb0 0x7f56f40a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f570c107c90 0x7f570c1a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f570c105360 0x7f570c19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 --2- 172.21.15.26:0/757408528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f570c102a30 0x7f570c19a470 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 >> 172.21.15.26:0/757408528 conn(0x7f570c0fc440 msgr2=0x7f570c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 shutdown_connections 2024-04-23T18:15:11.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.653+0000 7f5710e7b640 1 -- 172.21.15.26:0/757408528 wait complete. 2024-04-23T18:15:11.661 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-23T18:15:11.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-23T18:15:11.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 -- 172.21.15.26:0/94733549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 msgr2=0x7fdcb01077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 --2- 172.21.15.26:0/94733549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01077d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009900 tx=0x7fdca402a410 comp rx=0 tx=0).stop 2024-04-23T18:15:11.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 -- 172.21.15.26:0/94733549 shutdown_connections 2024-04-23T18:15:11.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 --2- 172.21.15.26:0/94733549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 0x7fdcb010a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 --2- 172.21.15.26:0/94733549 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01077d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 --2- 172.21.15.26:0/94733549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 0x7fdcb0104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 -- 172.21.15.26:0/94733549 >> 172.21.15.26:0/94733549 conn(0x7fdcb00fc400 msgr2=0x7fdcb00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 -- 172.21.15.26:0/94733549 shutdown_connections 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.780+0000 7fdcb5520640 1 -- 172.21.15.26:0/94733549 wait complete. 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 Processor -- start 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 -- start start 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 0x7fdcb0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 0x7fdcb019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb010daf0 con 0x7fdcb01053e0 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb010d970 con 0x7fdcb0107d10 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcb5520640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb010dc70 con 0x7fdcb0102ab0 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38076/0 (socket says 172.21.15.26:38076) 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcae7fc640 1 -- 172.21.15.26:0/2130206673 learned_addr learned my addr 172.21.15.26:0/2130206673 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.781+0000 7fdcaf7fe640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 0x7fdcb019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcaeffd640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 0x7fdcb0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 -- 172.21.15.26:0/2130206673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 msgr2=0x7fdcb0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 0x7fdcb0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 -- 172.21.15.26:0/2130206673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 msgr2=0x7fdcb019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 0x7fdcb019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 -- 172.21.15.26:0/2130206673 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb019fe90 con 0x7fdcb01053e0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcae7fc640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006010 tx=0x7fdca40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca403f030 con 0x7fdcb01053e0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb01a0120 con 0x7fdcb01053e0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdca4003cb0 con 0x7fdcb01053e0 2024-04-23T18:15:11.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4035070 con 0x7fdcb01053e0 2024-04-23T18:15:11.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.782+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb01a0610 con 0x7fdcb01053e0 2024-04-23T18:15:11.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.783+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb00fdef0 con 0x7fdcb01053e0 2024-04-23T18:15:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.784+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdca4042030 con 0x7fdcb01053e0 2024-04-23T18:15:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.785+0000 7fdc97fff640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 0x7fdc980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.785+0000 7fdcaeffd640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 0x7fdc980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.785+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdca40f0990 con 0x7fdcb01053e0 2024-04-23T18:15:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.785+0000 7fdcaeffd640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 0x7fdc980a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdca0005fd0 tx=0x7fdca0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:11.789+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdca40bd420 con 0x7fdcb01053e0 2024-04-23T18:15:11.942 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: 2024-04-23T18:15:11.549+0000 7f2b12977700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: 2024-04-23T18:15:11.549+0000 7f2b12977700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:active 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: Cluster is now healthy 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:15:11.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:active 2024-04-23T18:15:11.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:11 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:active 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: Cluster is now healthy 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:active 2024-04-23T18:15:11.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:11 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2961569454' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:15:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] up:active 2024-04-23T18:15:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:15:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:15:11.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: Cluster is now healthy 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/757408528' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] up:active 2024-04-23T18:15:11.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:11 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:15:11.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.004+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdcb0062390 con 0x7fdc980a3bb0 2024-04-23T18:15:12.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.027+0000 7fdc97fff640 1 -- 172.21.15.26:0/2130206673 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdcb0062390 con 0x7fdc980a3bb0 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 msgr2=0x7fdc980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 0x7fdc980a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdca0005fd0 tx=0x7fdca0005850 comp rx=0 tx=0).stop 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 msgr2=0x7fdcb01987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006010 tx=0x7fdca40042f0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 shutdown_connections 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdc980a3bb0 0x7fdc980a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcb0107d10 0x7fdcb019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdcb01053e0 0x7fdcb01987b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 --2- 172.21.15.26:0/2130206673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdcb0102ab0 0x7fdcb0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 >> 172.21.15.26:0/2130206673 conn(0x7fdcb00fc400 msgr2=0x7fdcb0103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 shutdown_connections 2024-04-23T18:15:12.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.029+0000 7fdcb5520640 1 -- 172.21.15.26:0/2130206673 wait complete. 2024-04-23T18:15:12.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.146+0000 7fb3a3508640 1 -- 172.21.15.26:0/2774434187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c1029a0 msgr2=0x7fb39c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.146+0000 7fb3a3508640 1 --2- 172.21.15.26:0/2774434187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c1029a0 0x7fb39c1092c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ca0 tx=0x7fb39402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- 172.21.15.26:0/2774434187 shutdown_connections 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- 172.21.15.26:0/2774434187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c101fe0 0x7fb39c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- 172.21.15.26:0/2774434187 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c1029a0 0x7fb39c1092c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- 172.21.15.26:0/2774434187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c100e90 0x7fb39c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- 172.21.15.26:0/2774434187 >> 172.21.15.26:0/2774434187 conn(0x7fb39c0fc400 msgr2=0x7fb39c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- 172.21.15.26:0/2774434187 shutdown_connections 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- 172.21.15.26:0/2774434187 wait complete. 2024-04-23T18:15:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 Processor -- start 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- start start 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 0x7fb39c1970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c1029a0 0x7fb39c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c10c9f0 con 0x7fb39c101fe0 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c10c870 con 0x7fb39c100e90 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.147+0000 7fb3a3508640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c10cb70 con 0x7fb39c1029a0 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38116/0 (socket says 172.21.15.26:38116) 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 -- 172.21.15.26:0/999420696 learned_addr learned my addr 172.21.15.26:0/999420696 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a1aa5640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c1029a0 0x7fb39c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a12a4640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 0x7fb39c1970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 -- 172.21.15.26:0/999420696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c1029a0 msgr2=0x7fb39c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c1029a0 0x7fb39c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 -- 172.21.15.26:0/999420696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 msgr2=0x7fb39c1970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 0x7fb39c1970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 -- 172.21.15.26:0/999420696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c19ecf0 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a12a4640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 0x7fb39c1970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a0aa3640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb39800d770 tx=0x7fb39800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398017070 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb398009c30 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398003cc0 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c19efe0 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.148+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c19f4a0 con 0x7fb39c101fe0 2024-04-23T18:15:12.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.149+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39c1081c0 con 0x7fb39c101fe0 2024-04-23T18:15:12.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.150+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb398003e60 con 0x7fb39c101fe0 2024-04-23T18:15:12.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.151+0000 7fb3927fc640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 0x7fb3880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.151+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb3980cfc50 con 0x7fb39c101fe0 2024-04-23T18:15:12.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.151+0000 7fb3a12a4640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 0x7fb3880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.151+0000 7fb3a12a4640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 0x7fb3880a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0058c0 tx=0x7fb38c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.156+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3980d5020 con 0x7fb39c101fe0 2024-04-23T18:15:12.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.369+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb39c062420 con 0x7fb3880a3bb0 2024-04-23T18:15:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.393+0000 7fb3927fc640 1 -- 172.21.15.26:0/999420696 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb39c062420 con 0x7fb3880a3bb0 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 msgr2=0x7fb3880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 0x7fb3880a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0058c0 tx=0x7fb38c005850 comp rx=0 tx=0).stop 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 msgr2=0x7fb39c197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb39800d770 tx=0x7fb39800dc30 comp rx=0 tx=0).stop 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 shutdown_connections 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3880a3bb0 0x7fb3880a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb39c1029a0 0x7fb39c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb39c101fe0 0x7fb39c197610 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 --2- 172.21.15.26:0/999420696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c100e90 0x7fb39c1970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 >> 172.21.15.26:0/999420696 conn(0x7fb39c0fc400 msgr2=0x7fb39c105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 shutdown_connections 2024-04-23T18:15:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.395+0000 7fb3a3508640 1 -- 172.21.15.26:0/999420696 wait complete. 2024-04-23T18:15:12.399 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:15:12.009+0000 7f57644e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:12.399 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:15:12.009+0000 7f57644e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:12.399 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:15:12.009+0000 7f57644e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:12.399 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:15:12.009+0000 7f57644e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:12.399 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:15:12.009+0000 7f57644e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:15:12.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-23T18:15:12.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 -- 172.21.15.26:0/1353349773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 msgr2=0x7fddac10a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 --2- 172.21.15.26:0/1353349773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac10a490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009cd0 tx=0x7fdda402a8e0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 -- 172.21.15.26:0/1353349773 shutdown_connections 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 --2- 172.21.15.26:0/1353349773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac10a490 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 --2- 172.21.15.26:0/1353349773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac103240 0x7fddac1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 --2- 172.21.15.26:0/1353349773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac100270 0x7fddac100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 -- 172.21.15.26:0/1353349773 >> 172.21.15.26:0/1353349773 conn(0x7fddac0fbb80 msgr2=0x7fddac0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 -- 172.21.15.26:0/1353349773 shutdown_connections 2024-04-23T18:15:12.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.513+0000 7fddb0820640 1 -- 172.21.15.26:0/1353349773 wait complete. 2024-04-23T18:15:12.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 Processor -- start 2024-04-23T18:15:12.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 -- start start 2024-04-23T18:15:12.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac100270 0x7fddac102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac103240 0x7fddac1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fddac10db10 con 0x7fddac103c00 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fddac10d990 con 0x7fddac103240 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.514+0000 7fddb0820640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fddac10dc90 con 0x7fddac100270 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38134/0 (socket says 172.21.15.26:38134) 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 -- 172.21.15.26:0/3108205335 learned_addr learned my addr 172.21.15.26:0/3108205335 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaa59c640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac100270 0x7fddac102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fdda9d9b640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac103240 0x7fddac1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 -- 172.21.15.26:0/3108205335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac100270 msgr2=0x7fddac102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac100270 0x7fddac102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 -- 172.21.15.26:0/3108205335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac103240 msgr2=0x7fddac1011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac103240 0x7fddac1011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 -- 172.21.15.26:0/3108205335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddac101fc0 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddaad9d640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdda40318b0 tx=0x7fdda4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4004240 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda40305a0 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddac1a22b0 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda40466e0 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.515+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddac1a2800 con 0x7fddac103c00 2024-04-23T18:15:12.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.516+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddac109370 con 0x7fddac103c00 2024-04-23T18:15:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.518+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdda4030750 con 0x7fddac103c00 2024-04-23T18:15:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.518+0000 7fdd9b7fe640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 0x7fdd940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.518+0000 7fddaa59c640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 0x7fdd940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.518+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdda402e030 con 0x7fddac103c00 2024-04-23T18:15:12.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.519+0000 7fddaa59c640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 0x7fdd940a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddac06a350 tx=0x7fdd9c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.522+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdda404b060 con 0x7fddac103c00 2024-04-23T18:15:12.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.738+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fddac062390 con 0x7fdd940a3bb0 2024-04-23T18:15:12.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.744+0000 7fdd9b7fe640 1 -- 172.21.15.26:0/3108205335 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fddac062390 con 0x7fdd940a3bb0 2024-04-23T18:15:12.738 INFO:teuthology.orchestra.run.smithi026.stdout:/volumes/qa/sv_0/e25be0af-b547-4cc0-afbf-4af3684e207b 2024-04-23T18:15:12.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 msgr2=0x7fdd940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 0x7fdd940a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddac06a350 tx=0x7fdd9c00a2d0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 msgr2=0x7fddac101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdda40318b0 tx=0x7fdda4003ba0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 shutdown_connections 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdd940a3bb0 0x7fdd940a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac103240 0x7fddac1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fddac103c00 0x7fddac101730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 --2- 172.21.15.26:0/3108205335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddac100270 0x7fddac102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 >> 172.21.15.26:0/3108205335 conn(0x7fddac0fbb80 msgr2=0x7fddac106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 shutdown_connections 2024-04-23T18:15:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.746+0000 7fddb0820640 1 -- 172.21.15.26:0/3108205335 wait complete. 2024-04-23T18:15:12.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 -- 172.21.15.26:0/1009714222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 msgr2=0x7f75ec1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 --2- 172.21.15.26:0/1009714222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec1024c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009900 tx=0x7f75dc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 -- 172.21.15.26:0/1009714222 shutdown_connections 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 --2- 172.21.15.26:0/1009714222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec102a00 0x7f75ec109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 --2- 172.21.15.26:0/1009714222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec1024c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 --2- 172.21.15.26:0/1009714222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec100e60 0x7f75ec101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 -- 172.21.15.26:0/1009714222 >> 172.21.15.26:0/1009714222 conn(0x7f75ec0fc420 msgr2=0x7f75ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 -- 172.21.15.26:0/1009714222 shutdown_connections 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.864+0000 7f75f4736640 1 -- 172.21.15.26:0/1009714222 wait complete. 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 Processor -- start 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 -- start start 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec100e60 0x7f75ec1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec102a00 0x7f75ec19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec10ca00 con 0x7f75ec102040 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec10c880 con 0x7f75ec102a00 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f4736640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec10cb80 con 0x7f75ec100e60 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38152/0 (socket says 172.21.15.26:38152) 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 -- 172.21.15.26:0/2756535762 learned_addr learned my addr 172.21.15.26:0/2756535762 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f24d2640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec100e60 0x7f75ec1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f2cd3640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec102a00 0x7f75ec19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 -- 172.21.15.26:0/2756535762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec100e60 msgr2=0x7f75ec1970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec100e60 0x7f75ec1970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 -- 172.21.15.26:0/2756535762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec102a00 msgr2=0x7f75ec19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec102a00 0x7f75ec19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 -- 172.21.15.26:0/2756535762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec19ed00 con 0x7f75ec102040 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.865+0000 7f75f1cd1640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75dc005940 tx=0x7f75dc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.866+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc03f030 con 0x7f75ec102040 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.866+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75dc003aa0 con 0x7f75ec102040 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.866+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec19ef90 con 0x7f75ec102040 2024-04-23T18:15:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.866+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc035070 con 0x7f75ec102040 2024-04-23T18:15:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.866+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec19f3f0 con 0x7f75ec102040 2024-04-23T18:15:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.867+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec108220 con 0x7f75ec102040 2024-04-23T18:15:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.868+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75dc042030 con 0x7f75ec102040 2024-04-23T18:15:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.869+0000 7f75e37fe640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 0x7f75d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.869+0000 7f75f24d2640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 0x7f75d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.869+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f75dc0f0da0 con 0x7f75ec102040 2024-04-23T18:15:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.869+0000 7f75f24d2640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 0x7f75d80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75ec101bb0 tx=0x7f75e800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:12.873+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75dc0bd280 con 0x7f75ec102040 2024-04-23T18:15:12.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:12 smithi026 ceph-mon[29364]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 13 op/s 2024-04-23T18:15:12.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:12 smithi131 ceph-mon[33041]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 13 op/s 2024-04-23T18:15:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.087+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f75ec062420 con 0x7f75d80a3bb0 2024-04-23T18:15:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.112+0000 7f75e37fe640 1 -- 172.21.15.26:0/2756535762 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f75ec062420 con 0x7f75d80a3bb0 2024-04-23T18:15:13.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 msgr2=0x7f75d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 0x7f75d80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75ec101bb0 tx=0x7f75e800b390 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 msgr2=0x7f75ec197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75dc005940 tx=0x7f75dc031a50 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 shutdown_connections 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f75d80a3bb0 0x7f75d80a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75ec102a00 0x7f75ec19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75ec102040 0x7f75ec197620 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 --2- 172.21.15.26:0/2756535762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec100e60 0x7f75ec1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 >> 172.21.15.26:0/2756535762 conn(0x7f75ec0fc420 msgr2=0x7f75ec1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 shutdown_connections 2024-04-23T18:15:13.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.115+0000 7f75f4736640 1 -- 172.21.15.26:0/2756535762 wait complete. 2024-04-23T18:15:13.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-23T18:15:13.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:12 smithi110 ceph-mon[32585]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 13 op/s 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.232+0000 7f8172919640 1 -- 172.21.15.26:0/3521123832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c1029a0 msgr2=0x7f816c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.232+0000 7f8172919640 1 --2- 172.21.15.26:0/3521123832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c1029a0 0x7f816c1092c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8164009ca0 tx=0x7f816402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 -- 172.21.15.26:0/3521123832 shutdown_connections 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- 172.21.15.26:0/3521123832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c101fe0 0x7f816c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- 172.21.15.26:0/3521123832 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c1029a0 0x7f816c1092c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- 172.21.15.26:0/3521123832 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c100e90 0x7f816c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 -- 172.21.15.26:0/3521123832 >> 172.21.15.26:0/3521123832 conn(0x7f816c0fc400 msgr2=0x7f816c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 -- 172.21.15.26:0/3521123832 shutdown_connections 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 -- 172.21.15.26:0/3521123832 wait complete. 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 Processor -- start 2024-04-23T18:15:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 -- start start 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c100e90 0x7f816c069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.233+0000 7f8172919640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c1029a0 0x7f816c1a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8172919640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f816c10cae0 con 0x7f816c101fe0 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8172919640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f816c10c960 con 0x7f816c100e90 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8172919640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f816c10cc60 con 0x7f816c1029a0 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38170/0 (socket says 172.21.15.26:38170) 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 -- 172.21.15.26:0/644575992 learned_addr learned my addr 172.21.15.26:0/644575992 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816bfff640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c100e90 0x7f816c069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8170eb6640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c1029a0 0x7f816c1a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 -- 172.21.15.26:0/644575992 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c1029a0 msgr2=0x7f816c1a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c1029a0 0x7f816c1a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 -- 172.21.15.26:0/644575992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c100e90 msgr2=0x7f816c069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c100e90 0x7f816c069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 -- 172.21.15.26:0/644575992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816c1a21e0 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f816b7fe640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f816000b450 tx=0x7f816000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160015020 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8160011d40 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816c1a2470 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816000f340 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.234+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f816c1a2a40 con 0x7f816c101fe0 2024-04-23T18:15:13.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.235+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f816c108310 con 0x7f816c101fe0 2024-04-23T18:15:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.236+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8160018020 con 0x7f816c101fe0 2024-04-23T18:15:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.237+0000 7f81697fa640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 0x7f81540a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.237+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f81600cfdc0 con 0x7f816c101fe0 2024-04-23T18:15:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.237+0000 7f816bfff640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 0x7f81540a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.237+0000 7f816bfff640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 0x7f81540a60a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f815c009890 tx=0x7f815c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.242+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f816001f020 con 0x7f816c101fe0 2024-04-23T18:15:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.459+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f816c062420 con 0x7f81540a3be0 2024-04-23T18:15:13.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.464+0000 7f81697fa640 1 -- 172.21.15.26:0/644575992 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f816c062420 con 0x7f81540a3be0 2024-04-23T18:15:13.458 INFO:teuthology.orchestra.run.smithi026.stdout:/volumes/qa/sv_1/f6f008e6-d2e3-40f3-a7ad-36a6f70dfaf6 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 msgr2=0x7f81540a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 0x7f81540a60a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f815c009890 tx=0x7f815c008040 comp rx=0 tx=0).stop 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 msgr2=0x7f816c19f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f816000b450 tx=0x7f816000b910 comp rx=0 tx=0).stop 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 shutdown_connections 2024-04-23T18:15:13.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f81540a3be0 0x7f81540a60a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f816c1029a0 0x7f816c1a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f816c101fe0 0x7f816c19f4a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 --2- 172.21.15.26:0/644575992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c100e90 0x7f816c069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 >> 172.21.15.26:0/644575992 conn(0x7f816c0fc400 msgr2=0x7f816c1054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 shutdown_connections 2024-04-23T18:15:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.466+0000 7f8172919640 1 -- 172.21.15.26:0/644575992 wait complete. 2024-04-23T18:15:13.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- 172.21.15.26:0/163817653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 msgr2=0x7f2c90104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 --2- 172.21.15.26:0/163817653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90104e80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009900 tx=0x7f2c8002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- 172.21.15.26:0/163817653 shutdown_connections 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 --2- 172.21.15.26:0/163817653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 0x7f2c901077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 --2- 172.21.15.26:0/163817653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 0x7f2c9010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 --2- 172.21.15.26:0/163817653 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90104e80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- 172.21.15.26:0/163817653 >> 172.21.15.26:0/163817653 conn(0x7f2c900fc440 msgr2=0x7f2c900fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- 172.21.15.26:0/163817653 shutdown_connections 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- 172.21.15.26:0/163817653 wait complete. 2024-04-23T18:15:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 Processor -- start 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.580+0000 7f2c960ae640 1 -- start start 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 0x7f2c90100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 0x7f2c90100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010daf0 con 0x7f2c90102a90 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010d970 con 0x7f2c901053c0 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c960ae640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010dc70 con 0x7f2c90107cf0 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38186/0 (socket says 172.21.15.26:38186) 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 -- 172.21.15.26:0/1922831343 learned_addr learned my addr 172.21.15.26:0/1922831343 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c86ffd640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 0x7f2c90100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8ffff640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 0x7f2c90100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 -- 172.21.15.26:0/1922831343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 msgr2=0x7f2c90100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 0x7f2c90100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 -- 172.21.15.26:0/1922831343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 msgr2=0x7f2c90100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 0x7f2c90100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 -- 172.21.15.26:0/1922831343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c90101810 con 0x7f2c90102a90 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8f7fe640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c80005940 tx=0x7f2c80004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.581+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8003f030 con 0x7f2c90102a90 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.582+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c80003cb0 con 0x7f2c90102a90 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.582+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c901a22f0 con 0x7f2c90102a90 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.582+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80035070 con 0x7f2c90102a90 2024-04-23T18:15:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.582+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c901a2840 con 0x7f2c90102a90 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.582+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c900fdf30 con 0x7f2c90102a90 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.584+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c80042030 con 0x7f2c90102a90 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.584+0000 7f2c8d7fa640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 0x7f2c7c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.584+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2c800f0cb0 con 0x7f2c90102a90 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.584+0000 7f2c86ffd640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 0x7f2c7c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.585+0000 7f2c86ffd640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 0x7f2c7c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c90102170 tx=0x7f2c7800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.589+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c80003e60 con 0x7f2c90102a90 2024-04-23T18:15:13.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.717+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2c8003bd00 con 0x7f2c90102a90 2024-04-23T18:15:13.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.821+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c9002d5c0 con 0x7f2c90102a90 2024-04-23T18:15:13.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.822+0000 7f2c8d7fa640 1 -- 172.21.15.26:0/1922831343 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f2c800bd7e0 con 0x7f2c90102a90 2024-04-23T18:15:13.816 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:13.816 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:15:13.816 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 msgr2=0x7f2c7c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 0x7f2c7c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c90102170 tx=0x7f2c7800a4c0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 msgr2=0x7f2c90102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c80005940 tx=0x7f2c80004430 comp rx=0 tx=0).stop 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 shutdown_connections 2024-04-23T18:15:13.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c7c0a3bb0 0x7f2c7c0a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c901053c0 0x7f2c90100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c90107cf0 0x7f2c90100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 --2- 172.21.15.26:0/1922831343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c90102a90 0x7f2c90102400 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 >> 172.21.15.26:0/1922831343 conn(0x7f2c900fc440 msgr2=0x7f2c901034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 shutdown_connections 2024-04-23T18:15:13.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.824+0000 7f2c960ae640 1 -- 172.21.15.26:0/1922831343 wait complete. 2024-04-23T18:15:13.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:13 smithi026 ceph-mon[29364]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:13.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:13 smithi026 ceph-mon[29364]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:13.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:13 smithi026 ceph-mon[29364]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:13.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:13.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.937+0000 7f2f1a664640 1 -- 172.21.15.26:0/3727530940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f14101fe0 msgr2=0x7f2f14102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.937+0000 7f2f1a664640 1 --2- 172.21.15.26:0/3727530940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f14101fe0 0x7f2f14102460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009900 tx=0x7f2efc02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.937+0000 7f2f1a664640 1 -- 172.21.15.26:0/3727530940 shutdown_connections 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- 172.21.15.26:0/3727530940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f141029a0 0x7f2f141092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- 172.21.15.26:0/3727530940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f14101fe0 0x7f2f14102460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- 172.21.15.26:0/3727530940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14100e90 0x7f2f141012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- 172.21.15.26:0/3727530940 >> 172.21.15.26:0/3727530940 conn(0x7f2f140fc400 msgr2=0x7f2f140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- 172.21.15.26:0/3727530940 shutdown_connections 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- 172.21.15.26:0/3727530940 wait complete. 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 Processor -- start 2024-04-23T18:15:13.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- start start 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f14100e90 0x7f2f141981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14101fe0 0x7f2f141986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1410c9d0 con 0x7f2f141029a0 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1410c850 con 0x7f2f14101fe0 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.938+0000 7f2f1a664640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1410cb50 con 0x7f2f14100e90 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38198/0 (socket says 172.21.15.26:38198) 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 -- 172.21.15.26:0/2109756516 learned_addr learned my addr 172.21.15.26:0/2109756516 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f13fff640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f14100e90 0x7f2f141981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f137fe640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14101fe0 0x7f2f141986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 -- 172.21.15.26:0/2109756516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f14100e90 msgr2=0x7f2f141981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f14100e90 0x7f2f141981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 -- 172.21.15.26:0/2109756516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14101fe0 msgr2=0x7f2f141986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14101fe0 0x7f2f141986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 -- 172.21.15.26:0/2109756516 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1419fdc0 con 0x7f2f141029a0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f18c01640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00cce0 tx=0x7f2f0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00f450 con 0x7f2f141029a0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f0c00faa0 con 0x7f2f141029a0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c018b70 con 0x7f2f141029a0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f141a00b0 con 0x7f2f141029a0 2024-04-23T18:15:13.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.939+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f141a0580 con 0x7f2f141029a0 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.940+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f141a0840 con 0x7f2f141029a0 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.942+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2f0c018d10 con 0x7f2f141029a0 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.942+0000 7f2f117fa640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 0x7f2f000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.942+0000 7f2f13fff640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 0x7f2f000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.942+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2f0c0d0780 con 0x7f2f141029a0 2024-04-23T18:15:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.943+0000 7f2f13fff640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 0x7f2f000a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400bbe0 tx=0x7f2f0400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:13.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:13.947+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f0c014030 con 0x7f2f141029a0 2024-04-23T18:15:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:13 smithi131 ceph-mon[33041]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:13 smithi131 ceph-mon[33041]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:13.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:13 smithi131 ceph-mon[33041]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.134+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f141a0a10 con 0x7f2f141029a0 2024-04-23T18:15:14.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.136+0000 7f2f117fa640 1 -- 172.21.15.26:0/2109756516 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7f2f0c09d1b0 con 0x7f2f141029a0 2024-04-23T18:15:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:14.130 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":86,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:15:07.425796+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:14.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 msgr2=0x7f2f000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 0x7f2f000a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400bbe0 tx=0x7f2f0400b390 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 msgr2=0x7f2f1419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00cce0 tx=0x7f2f0c00a620 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 shutdown_connections 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2f000a3c40 0x7f2f000a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f14101fe0 0x7f2f141986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f141029a0 0x7f2f1419f750 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 --2- 172.21.15.26:0/2109756516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f14100e90 0x7f2f141981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 >> 172.21.15.26:0/2109756516 conn(0x7f2f140fc400 msgr2=0x7f2f140fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 shutdown_connections 2024-04-23T18:15:14.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.138+0000 7f2f1a664640 1 -- 172.21.15.26:0/2109756516 wait complete. 2024-04-23T18:15:14.143 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T18:15:14.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:15:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:13 smithi110 ceph-mon[32585]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:13 smithi110 ceph-mon[32585]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:13 smithi110 ceph-mon[32585]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 -- 172.21.15.26:0/2138972876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8102a10 msgr2=0x7f27c8104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 --2- 172.21.15.26:0/2138972876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8102a10 0x7f27c8104e00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009900 tx=0x7f27b002a410 comp rx=0 tx=0).stop 2024-04-23T18:15:14.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 -- 172.21.15.26:0/2138972876 shutdown_connections 2024-04-23T18:15:14.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 --2- 172.21.15.26:0/2138972876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8107c70 0x7f27c810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 --2- 172.21.15.26:0/2138972876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 0x7f27c8107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 --2- 172.21.15.26:0/2138972876 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8102a10 0x7f27c8104e00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 -- 172.21.15.26:0/2138972876 >> 172.21.15.26:0/2138972876 conn(0x7f27c80fc400 msgr2=0x7f27c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 -- 172.21.15.26:0/2138972876 shutdown_connections 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.251+0000 7f27ce831640 1 -- 172.21.15.26:0/2138972876 wait complete. 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 Processor -- start 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 -- start start 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8102a10 0x7f27c8102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 0x7f27c81009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810db30 con 0x7f27c8107c70 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810d9b0 con 0x7f27c8102a10 2024-04-23T18:15:14.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ce831640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810dcb0 con 0x7f27c8105340 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.252+0000 7f27ccdce640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38224/0 (socket says 172.21.15.26:38224) 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 -- 172.21.15.26:0/3966787986 learned_addr learned my addr 172.21.15.26:0/3966787986 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27c7fff640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8102a10 0x7f27c8102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27c77fe640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 0x7f27c81009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 -- 172.21.15.26:0/3966787986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 msgr2=0x7f27c81009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 0x7f27c81009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 -- 172.21.15.26:0/3966787986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8102a10 msgr2=0x7f27c8102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8102a10 0x7f27c8102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 -- 172.21.15.26:0/3966787986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c8101790 con 0x7f27c8107c70 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ccdce640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27c000eaa0 tx=0x7f27c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c000cdb0 con 0x7f27c8107c70 2024-04-23T18:15:14.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c0005020 con 0x7f27c8107c70 2024-04-23T18:15:14.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c81a2380 con 0x7f27c8107c70 2024-04-23T18:15:14.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c00116d0 con 0x7f27c8107c70 2024-04-23T18:15:14.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.253+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c81a2900 con 0x7f27c8107c70 2024-04-23T18:15:14.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.254+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c80fdef0 con 0x7f27c8107c70 2024-04-23T18:15:14.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.255+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f27c00051d0 con 0x7f27c8107c70 2024-04-23T18:15:14.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.256+0000 7f27c57fa640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 0x7f27b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.256+0000 7f27c7fff640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 0x7f27b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.256+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f27c00d0720 con 0x7f27c8107c70 2024-04-23T18:15:14.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.256+0000 7f27c7fff640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 0x7f27b40a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f27b0000c00 tx=0x7f27b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.260+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27c0016030 con 0x7f27c8107c70 2024-04-23T18:15:14.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.443+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f27c8062390 con 0x7f27b40a3c40 2024-04-23T18:15:14.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.453+0000 7f27c57fa640 1 -- 172.21.15.26:0/3966787986 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258734 (secure 0 0 0) 0x7f27c8062390 con 0x7f27b40a3c40 2024-04-23T18:15:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:14.451 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:15:14.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.455+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 msgr2=0x7f27b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 0x7f27b40a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f27b0000c00 tx=0x7f27b00314a0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 msgr2=0x7f27c8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27c000eaa0 tx=0x7f27c000ef60 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 shutdown_connections 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f27b40a3c40 0x7f27b40a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f27c8107c70 0x7f27c8100f00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f27c8105340 0x7f27c81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 --2- 172.21.15.26:0/3966787986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27c8102a10 0x7f27c8102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 >> 172.21.15.26:0/3966787986 conn(0x7f27c80fc400 msgr2=0x7f27c8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 shutdown_connections 2024-04-23T18:15:14.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.456+0000 7f27ce831640 1 -- 172.21.15.26:0/3966787986 wait complete. 2024-04-23T18:15:14.461 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-04-23T18:15:12.838957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594274,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":281,"num_write_kb":2959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399400,"kb_used_data":12772,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124542424,"statfs":{"total":1151940427776,"available":1151531442176,"internally_reserved":0,"allocated":13078528,"data_stored":7100082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3906,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":100,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.166434"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713259+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:15:07.713259+0000","last_peered":"2024-04-23T18:15:07.713259+0000","last_clean":"2024-04-23T18:15:07.713259+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:15:07.713259+0000","last_undegraded":"2024-04-23T18:15:07.713259+0000","last_fullsized":"2024-04-23T18:15:07.713259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628203+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:15:07.628203+0000","last_peered":"2024-04-23T18:15:07.628203+0000","last_clean":"2024-04-23T18:15:07.628203+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:15:07.628203+0000","last_undegraded":"2024-04-23T18:15:07.628203+0000","last_fullsized":"2024-04-23T18:15:07.628203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627989+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:15:07.627989+0000","last_peered":"2024-04-23T18:15:07.627989+0000","last_clean":"2024-04-23T18:15:07.627989+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:15:07.627989+0000","last_undegraded":"2024-04-23T18:15:07.627989+0000","last_fullsized":"2024-04-23T18:15:07.627989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430243+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:15:07.430243+0000","last_peered":"2024-04-23T18:15:07.430243+0000","last_clean":"2024-04-23T18:15:07.430243+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:15:07.430243+0000","last_undegraded":"2024-04-23T18:15:07.430243+0000","last_fullsized":"2024-04-23T18:15:07.430243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628194+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:15:07.628194+0000","last_peered":"2024-04-23T18:15:07.628194+0000","last_clean":"2024-04-23T18:15:07.628194+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:15:07.628194+0000","last_undegraded":"2024-04-23T18:15:07.628194+0000","last_fullsized":"2024-04-23T18:15:07.628194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629613+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:15:07.629613+0000","last_peered":"2024-04-23T18:15:07.629613+0000","last_clean":"2024-04-23T18:15:07.629613+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:15:07.629613+0000","last_undegraded":"2024-04-23T18:15:07.629613+0000","last_fullsized":"2024-04-23T18:15:07.629613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628363+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:15:07.628363+0000","last_peered":"2024-04-23T18:15:07.628363+0000","last_clean":"2024-04-23T18:15:07.628363+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:15:07.628363+0000","last_undegraded":"2024-04-23T18:15:07.628363+0000","last_fullsized":"2024-04-23T18:15:07.628363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428885+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:15:07.428885+0000","last_peered":"2024-04-23T18:15:07.428885+0000","last_clean":"2024-04-23T18:15:07.428885+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:15:07.428885+0000","last_undegraded":"2024-04-23T18:15:07.428885+0000","last_fullsized":"2024-04-23T18:15:07.428885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629621+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:15:07.629621+0000","last_peered":"2024-04-23T18:15:07.629621+0000","last_clean":"2024-04-23T18:15:07.629621+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:15:07.629621+0000","last_undegraded":"2024-04-23T18:15:07.629621+0000","last_fullsized":"2024-04-23T18:15:07.629621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629681+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:15:07.629681+0000","last_peered":"2024-04-23T18:15:07.629681+0000","last_clean":"2024-04-23T18:15:07.629681+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:15:07.629681+0000","last_undegraded":"2024-04-23T18:15:07.629681+0000","last_fullsized":"2024-04-23T18:15:07.629681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629308+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:15:07.629308+0000","last_peered":"2024-04-23T18:15:07.629308+0000","last_clean":"2024-04-23T18:15:07.629308+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:15:07.629308+0000","last_undegraded":"2024-04-23T18:15:07.629308+0000","last_fullsized":"2024-04-23T18:15:07.629308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430178+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:15:07.430178+0000","last_peered":"2024-04-23T18:15:07.430178+0000","last_clean":"2024-04-23T18:15:07.430178+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:15:07.430178+0000","last_undegraded":"2024-04-23T18:15:07.430178+0000","last_fullsized":"2024-04-23T18:15:07.430178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460218+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:15:07.460218+0000","last_peered":"2024-04-23T18:15:07.460218+0000","last_clean":"2024-04-23T18:15:07.460218+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:15:07.460218+0000","last_undegraded":"2024-04-23T18:15:07.460218+0000","last_fullsized":"2024-04-23T18:15:07.460218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627763+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:15:07.627763+0000","last_peered":"2024-04-23T18:15:07.627763+0000","last_clean":"2024-04-23T18:15:07.627763+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:15:07.627763+0000","last_undegraded":"2024-04-23T18:15:07.627763+0000","last_fullsized":"2024-04-23T18:15:07.627763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627720+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:15:07.627720+0000","last_peered":"2024-04-23T18:15:07.627720+0000","last_clean":"2024-04-23T18:15:07.627720+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:15:07.627720+0000","last_undegraded":"2024-04-23T18:15:07.627720+0000","last_fullsized":"2024-04-23T18:15:07.627720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628593+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:15:07.628593+0000","last_peered":"2024-04-23T18:15:07.628593+0000","last_clean":"2024-04-23T18:15:07.628593+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:15:07.628593+0000","last_undegraded":"2024-04-23T18:15:07.628593+0000","last_fullsized":"2024-04-23T18:15:07.628593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713361+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:15:07.713361+0000","last_peered":"2024-04-23T18:15:07.713361+0000","last_clean":"2024-04-23T18:15:07.713361+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:15:07.713361+0000","last_undegraded":"2024-04-23T18:15:07.713361+0000","last_fullsized":"2024-04-23T18:15:07.713361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628078+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:15:07.628078+0000","last_peered":"2024-04-23T18:15:07.628078+0000","last_clean":"2024-04-23T18:15:07.628078+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:15:07.628078+0000","last_undegraded":"2024-04-23T18:15:07.628078+0000","last_fullsized":"2024-04-23T18:15:07.628078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628986+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:15:07.628986+0000","last_peered":"2024-04-23T18:15:07.628986+0000","last_clean":"2024-04-23T18:15:07.628986+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:15:07.628986+0000","last_undegraded":"2024-04-23T18:15:07.628986+0000","last_fullsized":"2024-04-23T18:15:07.628986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.710005+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:15:10.710005+0000","last_peered":"2024-04-23T18:15:10.710005+0000","last_clean":"2024-04-23T18:15:10.710005+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:15:10.710005+0000","last_undegraded":"2024-04-23T18:15:10.710005+0000","last_fullsized":"2024-04-23T18:15:10.710005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429431+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:15:07.429431+0000","last_peered":"2024-04-23T18:15:07.429431+0000","last_clean":"2024-04-23T18:15:07.429431+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:15:07.429431+0000","last_undegraded":"2024-04-23T18:15:07.429431+0000","last_fullsized":"2024-04-23T18:15:07.429431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429666+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:15:07.429666+0000","last_peered":"2024-04-23T18:15:07.429666+0000","last_clean":"2024-04-23T18:15:07.429666+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:15:07.429666+0000","last_undegraded":"2024-04-23T18:15:07.429666+0000","last_fullsized":"2024-04-23T18:15:07.429666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628439+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:15:07.628439+0000","last_peered":"2024-04-23T18:15:07.628439+0000","last_clean":"2024-04-23T18:15:07.628439+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:15:07.628439+0000","last_undegraded":"2024-04-23T18:15:07.628439+0000","last_fullsized":"2024-04-23T18:15:07.628439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629069+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:15:07.629069+0000","last_peered":"2024-04-23T18:15:07.629069+0000","last_clean":"2024-04-23T18:15:07.629069+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:15:07.629069+0000","last_undegraded":"2024-04-23T18:15:07.629069+0000","last_fullsized":"2024-04-23T18:15:07.629069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629219+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:15:07.629219+0000","last_peered":"2024-04-23T18:15:07.629219+0000","last_clean":"2024-04-23T18:15:07.629219+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:15:07.629219+0000","last_undegraded":"2024-04-23T18:15:07.629219+0000","last_fullsized":"2024-04-23T18:15:07.629219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628733+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:15:07.628733+0000","last_peered":"2024-04-23T18:15:07.628733+0000","last_clean":"2024-04-23T18:15:07.628733+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:15:07.628733+0000","last_undegraded":"2024-04-23T18:15:07.628733+0000","last_fullsized":"2024-04-23T18:15:07.628733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430351+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:15:07.430351+0000","last_peered":"2024-04-23T18:15:07.430351+0000","last_clean":"2024-04-23T18:15:07.430351+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:15:07.430351+0000","last_undegraded":"2024-04-23T18:15:07.430351+0000","last_fullsized":"2024-04-23T18:15:07.430351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:11.658678+0000","last_change":"2024-04-23T18:15:05.690635+0000","last_active":"2024-04-23T18:15:11.658678+0000","last_peered":"2024-04-23T18:15:11.658678+0000","last_clean":"2024-04-23T18:15:11.658678+0000","last_became_active":"2024-04-23T18:15:05.690068+0000","last_became_peered":"2024-04-23T18:15:05.690068+0000","last_unstale":"2024-04-23T18:15:11.658678+0000","last_undegraded":"2024-04-23T18:15:11.658678+0000","last_fullsized":"2024-04-23T18:15:11.658678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:10.334720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628230+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:15:07.628230+0000","last_peered":"2024-04-23T18:15:07.628230+0000","last_clean":"2024-04-23T18:15:07.628230+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:15:07.628230+0000","last_undegraded":"2024-04-23T18:15:07.628230+0000","last_fullsized":"2024-04-23T18:15:07.628230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.708924+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:15:10.708924+0000","last_peered":"2024-04-23T18:15:10.708924+0000","last_clean":"2024-04-23T18:15:10.708924+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:15:10.708924+0000","last_undegraded":"2024-04-23T18:15:10.708924+0000","last_fullsized":"2024-04-23T18:15:10.708924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430256+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:15:07.430256+0000","last_peered":"2024-04-23T18:15:07.430256+0000","last_clean":"2024-04-23T18:15:07.430256+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:15:07.430256+0000","last_undegraded":"2024-04-23T18:15:07.430256+0000","last_fullsized":"2024-04-23T18:15:07.430256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.707884+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:15:10.707884+0000","last_peered":"2024-04-23T18:15:10.707884+0000","last_clean":"2024-04-23T18:15:10.707884+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:15:10.707884+0000","last_undegraded":"2024-04-23T18:15:10.707884+0000","last_fullsized":"2024-04-23T18:15:10.707884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428978+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:15:07.428978+0000","last_peered":"2024-04-23T18:15:07.428978+0000","last_clean":"2024-04-23T18:15:07.428978+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:15:07.428978+0000","last_undegraded":"2024-04-23T18:15:07.428978+0000","last_fullsized":"2024-04-23T18:15:07.428978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.773266+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:15:07.773266+0000","last_peered":"2024-04-23T18:15:07.773266+0000","last_clean":"2024-04-23T18:15:07.773266+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:15:07.773266+0000","last_undegraded":"2024-04-23T18:15:07.773266+0000","last_fullsized":"2024-04-23T18:15:07.773266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628182+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:15:07.628182+0000","last_peered":"2024-04-23T18:15:07.628182+0000","last_clean":"2024-04-23T18:15:07.628182+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:15:07.628182+0000","last_undegraded":"2024-04-23T18:15:07.628182+0000","last_fullsized":"2024-04-23T18:15:07.628182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703457+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:15:09.703457+0000","last_peered":"2024-04-23T18:15:09.703457+0000","last_clean":"2024-04-23T18:15:09.703457+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:15:09.703457+0000","last_undegraded":"2024-04-23T18:15:09.703457+0000","last_fullsized":"2024-04-23T18:15:09.703457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629211+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:15:07.629211+0000","last_peered":"2024-04-23T18:15:07.629211+0000","last_clean":"2024-04-23T18:15:07.629211+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:15:07.629211+0000","last_undegraded":"2024-04-23T18:15:07.629211+0000","last_fullsized":"2024-04-23T18:15:07.629211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702889+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:15:09.702889+0000","last_peered":"2024-04-23T18:15:09.702889+0000","last_clean":"2024-04-23T18:15:09.702889+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:15:09.702889+0000","last_undegraded":"2024-04-23T18:15:09.702889+0000","last_fullsized":"2024-04-23T18:15:09.702889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628269+0000","last_change":"2024-04-23T18:15:06.450735+0000","last_active":"2024-04-23T18:15:07.628269+0000","last_peered":"2024-04-23T18:15:07.628269+0000","last_clean":"2024-04-23T18:15:07.628269+0000","last_became_active":"2024-04-23T18:15:06.450373+0000","last_became_peered":"2024-04-23T18:15:06.450373+0000","last_unstale":"2024-04-23T18:15:07.628269+0000","last_undegraded":"2024-04-23T18:15:07.628269+0000","last_fullsized":"2024-04-23T18:15:07.628269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:46.417279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703359+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:15:09.703359+0000","last_peered":"2024-04-23T18:15:09.703359+0000","last_clean":"2024-04-23T18:15:09.703359+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:15:09.703359+0000","last_undegraded":"2024-04-23T18:15:09.703359+0000","last_fullsized":"2024-04-23T18:15:09.703359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629465+0000","last_change":"2024-04-23T18:15:06.452530+0000","last_active":"2024-04-23T18:15:07.629465+0000","last_peered":"2024-04-23T18:15:07.629465+0000","last_clean":"2024-04-23T18:15:07.629465+0000","last_became_active":"2024-04-23T18:15:06.452192+0000","last_became_peered":"2024-04-23T18:15:06.452192+0000","last_unstale":"2024-04-23T18:15:07.629465+0000","last_undegraded":"2024-04-23T18:15:07.629465+0000","last_fullsized":"2024-04-23T18:15:07.629465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:16.563339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703782+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:15:09.703782+0000","last_peered":"2024-04-23T18:15:09.703782+0000","last_clean":"2024-04-23T18:15:09.703782+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:15:09.703782+0000","last_undegraded":"2024-04-23T18:15:09.703782+0000","last_fullsized":"2024-04-23T18:15:09.703782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629300+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:15:07.629300+0000","last_peered":"2024-04-23T18:15:07.629300+0000","last_clean":"2024-04-23T18:15:07.629300+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:15:07.629300+0000","last_undegraded":"2024-04-23T18:15:07.629300+0000","last_fullsized":"2024-04-23T18:15:07.629300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703967+0000","last_change":"2024-04-23T18:15:05.694655+0000","last_active":"2024-04-23T18:15:09.703967+0000","last_peered":"2024-04-23T18:15:09.703967+0000","last_clean":"2024-04-23T18:15:09.703967+0000","last_became_active":"2024-04-23T18:15:05.693960+0000","last_became_peered":"2024-04-23T18:15:05.693960+0000","last_unstale":"2024-04-23T18:15:09.703967+0000","last_undegraded":"2024-04-23T18:15:09.703967+0000","last_fullsized":"2024-04-23T18:15:09.703967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:25.720237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460136+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:15:07.460136+0000","last_peered":"2024-04-23T18:15:07.460136+0000","last_clean":"2024-04-23T18:15:07.460136+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:15:07.460136+0000","last_undegraded":"2024-04-23T18:15:07.460136+0000","last_fullsized":"2024-04-23T18:15:07.460136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703138+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:15:09.703138+0000","last_peered":"2024-04-23T18:15:09.703138+0000","last_clean":"2024-04-23T18:15:09.703138+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:15:09.703138+0000","last_undegraded":"2024-04-23T18:15:09.703138+0000","last_fullsized":"2024-04-23T18:15:09.703138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629009+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:15:07.629009+0000","last_peered":"2024-04-23T18:15:07.629009+0000","last_clean":"2024-04-23T18:15:07.629009+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:15:07.629009+0000","last_undegraded":"2024-04-23T18:15:07.629009+0000","last_fullsized":"2024-04-23T18:15:07.629009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429874+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:15:07.429874+0000","last_peered":"2024-04-23T18:15:07.429874+0000","last_clean":"2024-04-23T18:15:07.429874+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:15:07.429874+0000","last_undegraded":"2024-04-23T18:15:07.429874+0000","last_fullsized":"2024-04-23T18:15:07.429874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628318+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:15:07.628318+0000","last_peered":"2024-04-23T18:15:07.628318+0000","last_clean":"2024-04-23T18:15:07.628318+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:15:07.628318+0000","last_undegraded":"2024-04-23T18:15:07.628318+0000","last_fullsized":"2024-04-23T18:15:07.628318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702365+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:15:09.702365+0000","last_peered":"2024-04-23T18:15:09.702365+0000","last_clean":"2024-04-23T18:15:09.702365+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:15:09.702365+0000","last_undegraded":"2024-04-23T18:15:09.702365+0000","last_fullsized":"2024-04-23T18:15:09.702365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429843+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:15:07.429843+0000","last_peered":"2024-04-23T18:15:07.429843+0000","last_clean":"2024-04-23T18:15:07.429843+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:15:07.429843+0000","last_undegraded":"2024-04-23T18:15:07.429843+0000","last_fullsized":"2024-04-23T18:15:07.429843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628634+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:15:07.628634+0000","last_peered":"2024-04-23T18:15:07.628634+0000","last_clean":"2024-04-23T18:15:07.628634+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:15:07.628634+0000","last_undegraded":"2024-04-23T18:15:07.628634+0000","last_fullsized":"2024-04-23T18:15:07.628634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713021+0000","last_change":"2024-04-23T18:15:06.448781+0000","last_active":"2024-04-23T18:15:07.713021+0000","last_peered":"2024-04-23T18:15:07.713021+0000","last_clean":"2024-04-23T18:15:07.713021+0000","last_became_active":"2024-04-23T18:15:06.448403+0000","last_became_peered":"2024-04-23T18:15:06.448403+0000","last_unstale":"2024-04-23T18:15:07.713021+0000","last_undegraded":"2024-04-23T18:15:07.713021+0000","last_fullsized":"2024-04-23T18:15:07.713021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:35:38.094338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628345+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:15:07.628345+0000","last_peered":"2024-04-23T18:15:07.628345+0000","last_clean":"2024-04-23T18:15:07.628345+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:15:07.628345+0000","last_undegraded":"2024-04-23T18:15:07.628345+0000","last_fullsized":"2024-04-23T18:15:07.628345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629649+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:15:07.629649+0000","last_peered":"2024-04-23T18:15:07.629649+0000","last_clean":"2024-04-23T18:15:07.629649+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:15:07.629649+0000","last_undegraded":"2024-04-23T18:15:07.629649+0000","last_fullsized":"2024-04-23T18:15:07.629649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628988+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:15:07.628988+0000","last_peered":"2024-04-23T18:15:07.628988+0000","last_clean":"2024-04-23T18:15:07.628988+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:15:07.628988+0000","last_undegraded":"2024-04-23T18:15:07.628988+0000","last_fullsized":"2024-04-23T18:15:07.628988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629850+0000","last_change":"2024-04-23T18:15:06.451102+0000","last_active":"2024-04-23T18:15:07.629850+0000","last_peered":"2024-04-23T18:15:07.629850+0000","last_clean":"2024-04-23T18:15:07.629850+0000","last_became_active":"2024-04-23T18:15:06.450776+0000","last_became_peered":"2024-04-23T18:15:06.450776+0000","last_unstale":"2024-04-23T18:15:07.629850+0000","last_undegraded":"2024-04-23T18:15:07.629850+0000","last_fullsized":"2024-04-23T18:15:07.629850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:16:50.619819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429828+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:15:07.429828+0000","last_peered":"2024-04-23T18:15:07.429828+0000","last_clean":"2024-04-23T18:15:07.429828+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:15:07.429828+0000","last_undegraded":"2024-04-23T18:15:07.429828+0000","last_fullsized":"2024-04-23T18:15:07.429828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.709500+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:15:10.709500+0000","last_peered":"2024-04-23T18:15:10.709500+0000","last_clean":"2024-04-23T18:15:10.709500+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:15:10.709500+0000","last_undegraded":"2024-04-23T18:15:10.709500+0000","last_fullsized":"2024-04-23T18:15:10.709500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429550+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:15:07.429550+0000","last_peered":"2024-04-23T18:15:07.429550+0000","last_clean":"2024-04-23T18:15:07.429550+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:15:07.429550+0000","last_undegraded":"2024-04-23T18:15:07.429550+0000","last_fullsized":"2024-04-23T18:15:07.429550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430267+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:15:07.430267+0000","last_peered":"2024-04-23T18:15:07.430267+0000","last_clean":"2024-04-23T18:15:07.430267+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:15:07.430267+0000","last_undegraded":"2024-04-23T18:15:07.430267+0000","last_fullsized":"2024-04-23T18:15:07.430267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628532+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:15:07.628532+0000","last_peered":"2024-04-23T18:15:07.628532+0000","last_clean":"2024-04-23T18:15:07.628532+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:15:07.628532+0000","last_undegraded":"2024-04-23T18:15:07.628532+0000","last_fullsized":"2024-04-23T18:15:07.628532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460090+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:15:07.460090+0000","last_peered":"2024-04-23T18:15:07.460090+0000","last_clean":"2024-04-23T18:15:07.460090+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:15:07.460090+0000","last_undegraded":"2024-04-23T18:15:07.460090+0000","last_fullsized":"2024-04-23T18:15:07.460090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.708256+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:15:10.708256+0000","last_peered":"2024-04-23T18:15:10.708256+0000","last_clean":"2024-04-23T18:15:10.708256+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:15:10.708256+0000","last_undegraded":"2024-04-23T18:15:10.708256+0000","last_fullsized":"2024-04-23T18:15:10.708256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628902+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:15:07.628902+0000","last_peered":"2024-04-23T18:15:07.628902+0000","last_clean":"2024-04-23T18:15:07.628902+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:15:07.628902+0000","last_undegraded":"2024-04-23T18:15:07.628902+0000","last_fullsized":"2024-04-23T18:15:07.628902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460284+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:15:07.460284+0000","last_peered":"2024-04-23T18:15:07.460284+0000","last_clean":"2024-04-23T18:15:07.460284+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:15:07.460284+0000","last_undegraded":"2024-04-23T18:15:07.460284+0000","last_fullsized":"2024-04-23T18:15:07.460284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.709290+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:15:10.709290+0000","last_peered":"2024-04-23T18:15:10.709290+0000","last_clean":"2024-04-23T18:15:10.709290+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:15:10.709290+0000","last_undegraded":"2024-04-23T18:15:10.709290+0000","last_fullsized":"2024-04-23T18:15:10.709290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429290+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:15:07.429290+0000","last_peered":"2024-04-23T18:15:07.429290+0000","last_clean":"2024-04-23T18:15:07.429290+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:15:07.429290+0000","last_undegraded":"2024-04-23T18:15:07.429290+0000","last_fullsized":"2024-04-23T18:15:07.429290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430260+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:15:07.430260+0000","last_peered":"2024-04-23T18:15:07.430260+0000","last_clean":"2024-04-23T18:15:07.430260+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:15:07.430260+0000","last_undegraded":"2024-04-23T18:15:07.430260+0000","last_fullsized":"2024-04-23T18:15:07.430260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628373+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:15:07.628373+0000","last_peered":"2024-04-23T18:15:07.628373+0000","last_clean":"2024-04-23T18:15:07.628373+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:15:07.628373+0000","last_undegraded":"2024-04-23T18:15:07.628373+0000","last_fullsized":"2024-04-23T18:15:07.628373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703335+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:15:09.703335+0000","last_peered":"2024-04-23T18:15:09.703335+0000","last_clean":"2024-04-23T18:15:09.703335+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:15:09.703335+0000","last_undegraded":"2024-04-23T18:15:09.703335+0000","last_fullsized":"2024-04-23T18:15:09.703335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628281+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:15:07.628281+0000","last_peered":"2024-04-23T18:15:07.628281+0000","last_clean":"2024-04-23T18:15:07.628281+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:15:07.628281+0000","last_undegraded":"2024-04-23T18:15:07.628281+0000","last_fullsized":"2024-04-23T18:15:07.628281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702849+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:15:09.702849+0000","last_peered":"2024-04-23T18:15:09.702849+0000","last_clean":"2024-04-23T18:15:09.702849+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:15:09.702849+0000","last_undegraded":"2024-04-23T18:15:09.702849+0000","last_fullsized":"2024-04-23T18:15:09.702849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.712853+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:15:07.712853+0000","last_peered":"2024-04-23T18:15:07.712853+0000","last_clean":"2024-04-23T18:15:07.712853+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:15:07.712853+0000","last_undegraded":"2024-04-23T18:15:07.712853+0000","last_fullsized":"2024-04-23T18:15:07.712853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628417+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:15:07.628417+0000","last_peered":"2024-04-23T18:15:07.628417+0000","last_clean":"2024-04-23T18:15:07.628417+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:15:07.628417+0000","last_undegraded":"2024-04-23T18:15:07.628417+0000","last_fullsized":"2024-04-23T18:15:07.628417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429250+0000","last_change":"2024-04-23T18:15:06.445013+0000","last_active":"2024-04-23T18:15:07.429250+0000","last_peered":"2024-04-23T18:15:07.429250+0000","last_clean":"2024-04-23T18:15:07.429250+0000","last_became_active":"2024-04-23T18:15:06.443275+0000","last_became_peered":"2024-04-23T18:15:06.443275+0000","last_unstale":"2024-04-23T18:15:07.429250+0000","last_undegraded":"2024-04-23T18:15:07.429250+0000","last_fullsized":"2024-04-23T18:15:07.429250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:58.688700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703111+0000","last_change":"2024-04-23T18:15:05.695339+0000","last_active":"2024-04-23T18:15:09.703111+0000","last_peered":"2024-04-23T18:15:09.703111+0000","last_clean":"2024-04-23T18:15:09.703111+0000","last_became_active":"2024-04-23T18:15:05.695025+0000","last_became_peered":"2024-04-23T18:15:05.695025+0000","last_unstale":"2024-04-23T18:15:09.703111+0000","last_undegraded":"2024-04-23T18:15:09.703111+0000","last_fullsized":"2024-04-23T18:15:09.703111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:55.134576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629123+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:15:07.629123+0000","last_peered":"2024-04-23T18:15:07.629123+0000","last_clean":"2024-04-23T18:15:07.629123+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:15:07.629123+0000","last_undegraded":"2024-04-23T18:15:07.629123+0000","last_fullsized":"2024-04-23T18:15:07.629123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629020+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:15:07.629020+0000","last_peered":"2024-04-23T18:15:07.629020+0000","last_clean":"2024-04-23T18:15:07.629020+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:15:07.629020+0000","last_undegraded":"2024-04-23T18:15:07.629020+0000","last_fullsized":"2024-04-23T18:15:07.629020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629327+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:15:07.629327+0000","last_peered":"2024-04-23T18:15:07.629327+0000","last_clean":"2024-04-23T18:15:07.629327+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:15:07.629327+0000","last_undegraded":"2024-04-23T18:15:07.629327+0000","last_fullsized":"2024-04-23T18:15:07.629327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.773364+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:15:07.773364+0000","last_peered":"2024-04-23T18:15:07.773364+0000","last_clean":"2024-04-23T18:15:07.773364+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:15:07.773364+0000","last_undegraded":"2024-04-23T18:15:07.773364+0000","last_fullsized":"2024-04-23T18:15:07.773364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628534+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:15:07.628534+0000","last_peered":"2024-04-23T18:15:07.628534+0000","last_clean":"2024-04-23T18:15:07.628534+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:15:07.628534+0000","last_undegraded":"2024-04-23T18:15:07.628534+0000","last_fullsized":"2024-04-23T18:15:07.628534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460341+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:15:07.460341+0000","last_peered":"2024-04-23T18:15:07.460341+0000","last_clean":"2024-04-23T18:15:07.460341+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:15:07.460341+0000","last_undegraded":"2024-04-23T18:15:07.460341+0000","last_fullsized":"2024-04-23T18:15:07.460341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629144+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:15:07.629144+0000","last_peered":"2024-04-23T18:15:07.629144+0000","last_clean":"2024-04-23T18:15:07.629144+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:15:07.629144+0000","last_undegraded":"2024-04-23T18:15:07.629144+0000","last_fullsized":"2024-04-23T18:15:07.629144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713038+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:15:07.713038+0000","last_peered":"2024-04-23T18:15:07.713038+0000","last_clean":"2024-04-23T18:15:07.713038+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:15:07.713038+0000","last_undegraded":"2024-04-23T18:15:07.713038+0000","last_fullsized":"2024-04-23T18:15:07.713038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628554+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:15:07.628554+0000","last_peered":"2024-04-23T18:15:07.628554+0000","last_clean":"2024-04-23T18:15:07.628554+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:15:07.628554+0000","last_undegraded":"2024-04-23T18:15:07.628554+0000","last_fullsized":"2024-04-23T18:15:07.628554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628476+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:15:07.628476+0000","last_peered":"2024-04-23T18:15:07.628476+0000","last_clean":"2024-04-23T18:15:07.628476+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:15:07.628476+0000","last_undegraded":"2024-04-23T18:15:07.628476+0000","last_fullsized":"2024-04-23T18:15:07.628476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628089+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:15:07.628089+0000","last_peered":"2024-04-23T18:15:07.628089+0000","last_clean":"2024-04-23T18:15:07.628089+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:15:07.628089+0000","last_undegraded":"2024-04-23T18:15:07.628089+0000","last_fullsized":"2024-04-23T18:15:07.628089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460310+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:15:07.460310+0000","last_peered":"2024-04-23T18:15:07.460310+0000","last_clean":"2024-04-23T18:15:07.460310+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:15:07.460310+0000","last_undegraded":"2024-04-23T18:15:07.460310+0000","last_fullsized":"2024-04-23T18:15:07.460310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713058+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:15:07.713058+0000","last_peered":"2024-04-23T18:15:07.713058+0000","last_clean":"2024-04-23T18:15:07.713058+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:15:07.713058+0000","last_undegraded":"2024-04-23T18:15:07.713058+0000","last_fullsized":"2024-04-23T18:15:07.713058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629163+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:15:07.629163+0000","last_peered":"2024-04-23T18:15:07.629163+0000","last_clean":"2024-04-23T18:15:07.629163+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:15:07.629163+0000","last_undegraded":"2024-04-23T18:15:07.629163+0000","last_fullsized":"2024-04-23T18:15:07.629163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429132+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:15:07.429132+0000","last_peered":"2024-04-23T18:15:07.429132+0000","last_clean":"2024-04-23T18:15:07.429132+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:15:07.429132+0000","last_undegraded":"2024-04-23T18:15:07.429132+0000","last_fullsized":"2024-04-23T18:15:07.429132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429841+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:15:07.429841+0000","last_peered":"2024-04-23T18:15:07.429841+0000","last_clean":"2024-04-23T18:15:07.429841+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:15:07.429841+0000","last_undegraded":"2024-04-23T18:15:07.429841+0000","last_fullsized":"2024-04-23T18:15:07.429841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428961+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:15:07.428961+0000","last_peered":"2024-04-23T18:15:07.428961+0000","last_clean":"2024-04-23T18:15:07.428961+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:15:07.428961+0000","last_undegraded":"2024-04-23T18:15:07.428961+0000","last_fullsized":"2024-04-23T18:15:07.428961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628923+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:15:07.628923+0000","last_peered":"2024-04-23T18:15:07.628923+0000","last_clean":"2024-04-23T18:15:07.628923+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:15:07.628923+0000","last_undegraded":"2024-04-23T18:15:07.628923+0000","last_fullsized":"2024-04-23T18:15:07.628923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460394+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:15:07.460394+0000","last_peered":"2024-04-23T18:15:07.460394+0000","last_clean":"2024-04-23T18:15:07.460394+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:15:07.460394+0000","last_undegraded":"2024-04-23T18:15:07.460394+0000","last_fullsized":"2024-04-23T18:15:07.460394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713242+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:15:07.713242+0000","last_peered":"2024-04-23T18:15:07.713242+0000","last_clean":"2024-04-23T18:15:07.713242+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:15:07.713242+0000","last_undegraded":"2024-04-23T18:15:07.713242+0000","last_fullsized":"2024-04-23T18:15:07.713242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3906,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":100,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":14148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":443015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1035951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":942080,"data_stored":443799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":443619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40996,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704156,"statfs":{"total":95995035648,"available":95953055744,"internally_reserved":0,"allocated":1536000,"data_stored":1035175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":443105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40428,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704724,"statfs":{"total":95995035648,"available":95953637376,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":445283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":443119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40416,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704736,"statfs":{"total":95995035648,"available":95953649664,"internally_reserved":0,"allocated":929792,"data_stored":443037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32688,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712464,"statfs":{"total":95995035648,"available":95961563136,"internally_reserved":0,"allocated":1548288,"data_stored":1034859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:15:14.466 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 -- 172.21.15.26:0/2916537330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf281029a0 msgr2=0x7fbf281092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/2916537330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf281029a0 0x7fbf281092c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009ca0 tx=0x7fbf2002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 -- 172.21.15.26:0/2916537330 shutdown_connections 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/2916537330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf28101fe0 0x7fbf28102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/2916537330 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf281029a0 0x7fbf281092c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/2916537330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 -- 172.21.15.26:0/2916537330 >> 172.21.15.26:0/2916537330 conn(0x7fbf280fc400 msgr2=0x7fbf280fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 -- 172.21.15.26:0/2916537330 shutdown_connections 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.610+0000 7fbf2cefd640 1 -- 172.21.15.26:0/2916537330 wait complete. 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 Processor -- start 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 -- start start 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 0x7fbf2819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810dac0 con 0x7fbf28101fe0 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810d940 con 0x7fbf281029a0 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2cefd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810dc40 con 0x7fbf28100e90 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38252/0 (socket says 172.21.15.26:38252) 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 -- 172.21.15.26:0/1420938351 learned_addr learned my addr 172.21.15.26:0/1420938351 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2659c640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 -- 172.21.15.26:0/1420938351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 msgr2=0x7fbf281980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf26d9d640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 0x7fbf2819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 -- 172.21.15.26:0/1420938351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 msgr2=0x7fbf2819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 0x7fbf2819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 -- 172.21.15.26:0/1420938351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2819fcf0 con 0x7fbf28101fe0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf2659c640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf26d9d640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 0x7fbf2819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.611+0000 7fbf25d9b640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00d770 tx=0x7fbf1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.612+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c017070 con 0x7fbf28101fe0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.612+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2819ffe0 con 0x7fbf28101fe0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.612+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf1c009c30 con 0x7fbf28101fe0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.612+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003cc0 con 0x7fbf28101fe0 2024-04-23T18:15:14.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.612+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf281a04a0 con 0x7fbf28101fe0 2024-04-23T18:15:14.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.613+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf28108140 con 0x7fbf28101fe0 2024-04-23T18:15:14.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.614+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbf1c003e60 con 0x7fbf28101fe0 2024-04-23T18:15:14.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.614+0000 7fbf177fe640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 0x7fbf100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:14.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.615+0000 7fbf2659c640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 0x7fbf100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:14.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.615+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fbf1c0cfe30 con 0x7fbf28101fe0 2024-04-23T18:15:14.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.615+0000 7fbf2659c640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 0x7fbf100a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800abe0 tx=0x7fbf1800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:14.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.619+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf1c09c860 con 0x7fbf28101fe0 2024-04-23T18:15:14.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.809+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf28062390 con 0x7fbf100a3c40 2024-04-23T18:15:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.817+0000 7fbf177fe640 1 -- 172.21.15.26:0/1420938351 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258734 (secure 0 0 0) 0x7fbf28062390 con 0x7fbf100a3c40 2024-04-23T18:15:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:15:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 msgr2=0x7fbf100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 0x7fbf100a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800abe0 tx=0x7fbf1800a5c0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 msgr2=0x7fbf28198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00d770 tx=0x7fbf1c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 shutdown_connections 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fbf100a3c40 0x7fbf100a6100 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf281029a0 0x7fbf2819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbf28101fe0 0x7fbf28198610 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 --2- 172.21.15.26:0/1420938351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf28100e90 0x7fbf281980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 >> 172.21.15.26:0/1420938351 conn(0x7fbf280fc400 msgr2=0x7fbf28105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 shutdown_connections 2024-04-23T18:15:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:14.820+0000 7fbf2cefd640 1 -- 172.21.15.26:0/1420938351 wait complete. 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: pgmap v287: 97 pgs: 97 active+clean; 580 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 714 B/s rd, 2.9 KiB/s wr, 14 op/s 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1922831343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2109756516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.825 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-04-23T18:15:12.838957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594274,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":281,"num_write_kb":2959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399400,"kb_used_data":12772,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124542424,"statfs":{"total":1151940427776,"available":1151531442176,"internally_reserved":0,"allocated":13078528,"data_stored":7100082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3906,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":100,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.166434"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713259+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:15:07.713259+0000","last_peered":"2024-04-23T18:15:07.713259+0000","last_clean":"2024-04-23T18:15:07.713259+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:15:07.713259+0000","last_undegraded":"2024-04-23T18:15:07.713259+0000","last_fullsized":"2024-04-23T18:15:07.713259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628203+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:15:07.628203+0000","last_peered":"2024-04-23T18:15:07.628203+0000","last_clean":"2024-04-23T18:15:07.628203+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:15:07.628203+0000","last_undegraded":"2024-04-23T18:15:07.628203+0000","last_fullsized":"2024-04-23T18:15:07.628203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627989+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:15:07.627989+0000","last_peered":"2024-04-23T18:15:07.627989+0000","last_clean":"2024-04-23T18:15:07.627989+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:15:07.627989+0000","last_undegraded":"2024-04-23T18:15:07.627989+0000","last_fullsized":"2024-04-23T18:15:07.627989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430243+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:15:07.430243+0000","last_peered":"2024-04-23T18:15:07.430243+0000","last_clean":"2024-04-23T18:15:07.430243+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:15:07.430243+0000","last_undegraded":"2024-04-23T18:15:07.430243+0000","last_fullsized":"2024-04-23T18:15:07.430243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628194+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:15:07.628194+0000","last_peered":"2024-04-23T18:15:07.628194+0000","last_clean":"2024-04-23T18:15:07.628194+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:15:07.628194+0000","last_undegraded":"2024-04-23T18:15:07.628194+0000","last_fullsized":"2024-04-23T18:15:07.628194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629613+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:15:07.629613+0000","last_peered":"2024-04-23T18:15:07.629613+0000","last_clean":"2024-04-23T18:15:07.629613+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:15:07.629613+0000","last_undegraded":"2024-04-23T18:15:07.629613+0000","last_fullsized":"2024-04-23T18:15:07.629613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628363+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:15:07.628363+0000","last_peered":"2024-04-23T18:15:07.628363+0000","last_clean":"2024-04-23T18:15:07.628363+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:15:07.628363+0000","last_undegraded":"2024-04-23T18:15:07.628363+0000","last_fullsized":"2024-04-23T18:15:07.628363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428885+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:15:07.428885+0000","last_peered":"2024-04-23T18:15:07.428885+0000","last_clean":"2024-04-23T18:15:07.428885+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:15:07.428885+0000","last_undegraded":"2024-04-23T18:15:07.428885+0000","last_fullsized":"2024-04-23T18:15:07.428885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629621+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:15:07.629621+0000","last_peered":"2024-04-23T18:15:07.629621+0000","last_clean":"2024-04-23T18:15:07.629621+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:15:07.629621+0000","last_undegraded":"2024-04-23T18:15:07.629621+0000","last_fullsized":"2024-04-23T18:15:07.629621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629681+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:15:07.629681+0000","last_peered":"2024-04-23T18:15:07.629681+0000","last_clean":"2024-04-23T18:15:07.629681+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:15:07.629681+0000","last_undegraded":"2024-04-23T18:15:07.629681+0000","last_fullsized":"2024-04-23T18:15:07.629681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629308+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:15:07.629308+0000","last_peered":"2024-04-23T18:15:07.629308+0000","last_clean":"2024-04-23T18:15:07.629308+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:15:07.629308+0000","last_undegraded":"2024-04-23T18:15:07.629308+0000","last_fullsized":"2024-04-23T18:15:07.629308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430178+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:15:07.430178+0000","last_peered":"2024-04-23T18:15:07.430178+0000","last_clean":"2024-04-23T18:15:07.430178+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:15:07.430178+0000","last_undegraded":"2024-04-23T18:15:07.430178+0000","last_fullsized":"2024-04-23T18:15:07.430178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460218+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:15:07.460218+0000","last_peered":"2024-04-23T18:15:07.460218+0000","last_clean":"2024-04-23T18:15:07.460218+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:15:07.460218+0000","last_undegraded":"2024-04-23T18:15:07.460218+0000","last_fullsized":"2024-04-23T18:15:07.460218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627763+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:15:07.627763+0000","last_peered":"2024-04-23T18:15:07.627763+0000","last_clean":"2024-04-23T18:15:07.627763+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:15:07.627763+0000","last_undegraded":"2024-04-23T18:15:07.627763+0000","last_fullsized":"2024-04-23T18:15:07.627763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.627720+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:15:07.627720+0000","last_peered":"2024-04-23T18:15:07.627720+0000","last_clean":"2024-04-23T18:15:07.627720+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:15:07.627720+0000","last_undegraded":"2024-04-23T18:15:07.627720+0000","last_fullsized":"2024-04-23T18:15:07.627720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628593+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:15:07.628593+0000","last_peered":"2024-04-23T18:15:07.628593+0000","last_clean":"2024-04-23T18:15:07.628593+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:15:07.628593+0000","last_undegraded":"2024-04-23T18:15:07.628593+0000","last_fullsized":"2024-04-23T18:15:07.628593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713361+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:15:07.713361+0000","last_peered":"2024-04-23T18:15:07.713361+0000","last_clean":"2024-04-23T18:15:07.713361+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:15:07.713361+0000","last_undegraded":"2024-04-23T18:15:07.713361+0000","last_fullsized":"2024-04-23T18:15:07.713361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628078+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:15:07.628078+0000","last_peered":"2024-04-23T18:15:07.628078+0000","last_clean":"2024-04-23T18:15:07.628078+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:15:07.628078+0000","last_undegraded":"2024-04-23T18:15:07.628078+0000","last_fullsized":"2024-04-23T18:15:07.628078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628986+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:15:07.628986+0000","last_peered":"2024-04-23T18:15:07.628986+0000","last_clean":"2024-04-23T18:15:07.628986+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:15:07.628986+0000","last_undegraded":"2024-04-23T18:15:07.628986+0000","last_fullsized":"2024-04-23T18:15:07.628986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.710005+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:15:10.710005+0000","last_peered":"2024-04-23T18:15:10.710005+0000","last_clean":"2024-04-23T18:15:10.710005+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:15:10.710005+0000","last_undegraded":"2024-04-23T18:15:10.710005+0000","last_fullsized":"2024-04-23T18:15:10.710005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429431+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:15:07.429431+0000","last_peered":"2024-04-23T18:15:07.429431+0000","last_clean":"2024-04-23T18:15:07.429431+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:15:07.429431+0000","last_undegraded":"2024-04-23T18:15:07.429431+0000","last_fullsized":"2024-04-23T18:15:07.429431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429666+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:15:07.429666+0000","last_peered":"2024-04-23T18:15:07.429666+0000","last_clean":"2024-04-23T18:15:07.429666+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:15:07.429666+0000","last_undegraded":"2024-04-23T18:15:07.429666+0000","last_fullsized":"2024-04-23T18:15:07.429666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628439+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:15:07.628439+0000","last_peered":"2024-04-23T18:15:07.628439+0000","last_clean":"2024-04-23T18:15:07.628439+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:15:07.628439+0000","last_undegraded":"2024-04-23T18:15:07.628439+0000","last_fullsized":"2024-04-23T18:15:07.628439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629069+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:15:07.629069+0000","last_peered":"2024-04-23T18:15:07.629069+0000","last_clean":"2024-04-23T18:15:07.629069+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:15:07.629069+0000","last_undegraded":"2024-04-23T18:15:07.629069+0000","last_fullsized":"2024-04-23T18:15:07.629069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629219+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:15:07.629219+0000","last_peered":"2024-04-23T18:15:07.629219+0000","last_clean":"2024-04-23T18:15:07.629219+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:15:07.629219+0000","last_undegraded":"2024-04-23T18:15:07.629219+0000","last_fullsized":"2024-04-23T18:15:07.629219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628733+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:15:07.628733+0000","last_peered":"2024-04-23T18:15:07.628733+0000","last_clean":"2024-04-23T18:15:07.628733+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:15:07.628733+0000","last_undegraded":"2024-04-23T18:15:07.628733+0000","last_fullsized":"2024-04-23T18:15:07.628733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430351+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:15:07.430351+0000","last_peered":"2024-04-23T18:15:07.430351+0000","last_clean":"2024-04-23T18:15:07.430351+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:15:07.430351+0000","last_undegraded":"2024-04-23T18:15:07.430351+0000","last_fullsized":"2024-04-23T18:15:07.430351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:11.658678+0000","last_change":"2024-04-23T18:15:05.690635+0000","last_active":"2024-04-23T18:15:11.658678+0000","last_peered":"2024-04-23T18:15:11.658678+0000","last_clean":"2024-04-23T18:15:11.658678+0000","last_became_active":"2024-04-23T18:15:05.690068+0000","last_became_peered":"2024-04-23T18:15:05.690068+0000","last_unstale":"2024-04-23T18:15:11.658678+0000","last_undegraded":"2024-04-23T18:15:11.658678+0000","last_fullsized":"2024-04-23T18:15:11.658678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:10.334720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628230+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:15:07.628230+0000","last_peered":"2024-04-23T18:15:07.628230+0000","last_clean":"2024-04-23T18:15:07.628230+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:15:07.628230+0000","last_undegraded":"2024-04-23T18:15:07.628230+0000","last_fullsized":"2024-04-23T18:15:07.628230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.708924+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:15:10.708924+0000","last_peered":"2024-04-23T18:15:10.708924+0000","last_clean":"2024-04-23T18:15:10.708924+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:15:10.708924+0000","last_undegraded":"2024-04-23T18:15:10.708924+0000","last_fullsized":"2024-04-23T18:15:10.708924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430256+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:15:07.430256+0000","last_peered":"2024-04-23T18:15:07.430256+0000","last_clean":"2024-04-23T18:15:07.430256+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:15:07.430256+0000","last_undegraded":"2024-04-23T18:15:07.430256+0000","last_fullsized":"2024-04-23T18:15:07.430256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.707884+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:15:10.707884+0000","last_peered":"2024-04-23T18:15:10.707884+0000","last_clean":"2024-04-23T18:15:10.707884+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:15:10.707884+0000","last_undegraded":"2024-04-23T18:15:10.707884+0000","last_fullsized":"2024-04-23T18:15:10.707884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428978+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:15:07.428978+0000","last_peered":"2024-04-23T18:15:07.428978+0000","last_clean":"2024-04-23T18:15:07.428978+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:15:07.428978+0000","last_undegraded":"2024-04-23T18:15:07.428978+0000","last_fullsized":"2024-04-23T18:15:07.428978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.773266+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:15:07.773266+0000","last_peered":"2024-04-23T18:15:07.773266+0000","last_clean":"2024-04-23T18:15:07.773266+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:15:07.773266+0000","last_undegraded":"2024-04-23T18:15:07.773266+0000","last_fullsized":"2024-04-23T18:15:07.773266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628182+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:15:07.628182+0000","last_peered":"2024-04-23T18:15:07.628182+0000","last_clean":"2024-04-23T18:15:07.628182+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:15:07.628182+0000","last_undegraded":"2024-04-23T18:15:07.628182+0000","last_fullsized":"2024-04-23T18:15:07.628182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703457+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:15:09.703457+0000","last_peered":"2024-04-23T18:15:09.703457+0000","last_clean":"2024-04-23T18:15:09.703457+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:15:09.703457+0000","last_undegraded":"2024-04-23T18:15:09.703457+0000","last_fullsized":"2024-04-23T18:15:09.703457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629211+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:15:07.629211+0000","last_peered":"2024-04-23T18:15:07.629211+0000","last_clean":"2024-04-23T18:15:07.629211+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:15:07.629211+0000","last_undegraded":"2024-04-23T18:15:07.629211+0000","last_fullsized":"2024-04-23T18:15:07.629211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702889+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:15:09.702889+0000","last_peered":"2024-04-23T18:15:09.702889+0000","last_clean":"2024-04-23T18:15:09.702889+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:15:09.702889+0000","last_undegraded":"2024-04-23T18:15:09.702889+0000","last_fullsized":"2024-04-23T18:15:09.702889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628269+0000","last_change":"2024-04-23T18:15:06.450735+0000","last_active":"2024-04-23T18:15:07.628269+0000","last_peered":"2024-04-23T18:15:07.628269+0000","last_clean":"2024-04-23T18:15:07.628269+0000","last_became_active":"2024-04-23T18:15:06.450373+0000","last_became_peered":"2024-04-23T18:15:06.450373+0000","last_unstale":"2024-04-23T18:15:07.628269+0000","last_undegraded":"2024-04-23T18:15:07.628269+0000","last_fullsized":"2024-04-23T18:15:07.628269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:46.417279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703359+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:15:09.703359+0000","last_peered":"2024-04-23T18:15:09.703359+0000","last_clean":"2024-04-23T18:15:09.703359+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:15:09.703359+0000","last_undegraded":"2024-04-23T18:15:09.703359+0000","last_fullsized":"2024-04-23T18:15:09.703359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629465+0000","last_change":"2024-04-23T18:15:06.452530+0000","last_active":"2024-04-23T18:15:07.629465+0000","last_peered":"2024-04-23T18:15:07.629465+0000","last_clean":"2024-04-23T18:15:07.629465+0000","last_became_active":"2024-04-23T18:15:06.452192+0000","last_became_peered":"2024-04-23T18:15:06.452192+0000","last_unstale":"2024-04-23T18:15:07.629465+0000","last_undegraded":"2024-04-23T18:15:07.629465+0000","last_fullsized":"2024-04-23T18:15:07.629465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:21:16.563339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703782+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:15:09.703782+0000","last_peered":"2024-04-23T18:15:09.703782+0000","last_clean":"2024-04-23T18:15:09.703782+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:15:09.703782+0000","last_undegraded":"2024-04-23T18:15:09.703782+0000","last_fullsized":"2024-04-23T18:15:09.703782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629300+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:15:07.629300+0000","last_peered":"2024-04-23T18:15:07.629300+0000","last_clean":"2024-04-23T18:15:07.629300+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:15:07.629300+0000","last_undegraded":"2024-04-23T18:15:07.629300+0000","last_fullsized":"2024-04-23T18:15:07.629300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703967+0000","last_change":"2024-04-23T18:15:05.694655+0000","last_active":"2024-04-23T18:15:09.703967+0000","last_peered":"2024-04-23T18:15:09.703967+0000","last_clean":"2024-04-23T18:15:09.703967+0000","last_became_active":"2024-04-23T18:15:05.693960+0000","last_became_peered":"2024-04-23T18:15:05.693960+0000","last_unstale":"2024-04-23T18:15:09.703967+0000","last_undegraded":"2024-04-23T18:15:09.703967+0000","last_fullsized":"2024-04-23T18:15:09.703967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:25.720237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460136+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:15:07.460136+0000","last_peered":"2024-04-23T18:15:07.460136+0000","last_clean":"2024-04-23T18:15:07.460136+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:15:07.460136+0000","last_undegraded":"2024-04-23T18:15:07.460136+0000","last_fullsized":"2024-04-23T18:15:07.460136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703138+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:15:09.703138+0000","last_peered":"2024-04-23T18:15:09.703138+0000","last_clean":"2024-04-23T18:15:09.703138+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:15:09.703138+0000","last_undegraded":"2024-04-23T18:15:09.703138+0000","last_fullsized":"2024-04-23T18:15:09.703138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629009+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:15:07.629009+0000","last_peered":"2024-04-23T18:15:07.629009+0000","last_clean":"2024-04-23T18:15:07.629009+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:15:07.629009+0000","last_undegraded":"2024-04-23T18:15:07.629009+0000","last_fullsized":"2024-04-23T18:15:07.629009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429874+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:15:07.429874+0000","last_peered":"2024-04-23T18:15:07.429874+0000","last_clean":"2024-04-23T18:15:07.429874+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:15:07.429874+0000","last_undegraded":"2024-04-23T18:15:07.429874+0000","last_fullsized":"2024-04-23T18:15:07.429874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628318+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:15:07.628318+0000","last_peered":"2024-04-23T18:15:07.628318+0000","last_clean":"2024-04-23T18:15:07.628318+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:15:07.628318+0000","last_undegraded":"2024-04-23T18:15:07.628318+0000","last_fullsized":"2024-04-23T18:15:07.628318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702365+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:15:09.702365+0000","last_peered":"2024-04-23T18:15:09.702365+0000","last_clean":"2024-04-23T18:15:09.702365+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:15:09.702365+0000","last_undegraded":"2024-04-23T18:15:09.702365+0000","last_fullsized":"2024-04-23T18:15:09.702365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429843+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:15:07.429843+0000","last_peered":"2024-04-23T18:15:07.429843+0000","last_clean":"2024-04-23T18:15:07.429843+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:15:07.429843+0000","last_undegraded":"2024-04-23T18:15:07.429843+0000","last_fullsized":"2024-04-23T18:15:07.429843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628634+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:15:07.628634+0000","last_peered":"2024-04-23T18:15:07.628634+0000","last_clean":"2024-04-23T18:15:07.628634+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:15:07.628634+0000","last_undegraded":"2024-04-23T18:15:07.628634+0000","last_fullsized":"2024-04-23T18:15:07.628634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713021+0000","last_change":"2024-04-23T18:15:06.448781+0000","last_active":"2024-04-23T18:15:07.713021+0000","last_peered":"2024-04-23T18:15:07.713021+0000","last_clean":"2024-04-23T18:15:07.713021+0000","last_became_active":"2024-04-23T18:15:06.448403+0000","last_became_peered":"2024-04-23T18:15:06.448403+0000","last_unstale":"2024-04-23T18:15:07.713021+0000","last_undegraded":"2024-04-23T18:15:07.713021+0000","last_fullsized":"2024-04-23T18:15:07.713021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:35:38.094338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628345+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:15:07.628345+0000","last_peered":"2024-04-23T18:15:07.628345+0000","last_clean":"2024-04-23T18:15:07.628345+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:15:07.628345+0000","last_undegraded":"2024-04-23T18:15:07.628345+0000","last_fullsized":"2024-04-23T18:15:07.628345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629649+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:15:07.629649+0000","last_peered":"2024-04-23T18:15:07.629649+0000","last_clean":"2024-04-23T18:15:07.629649+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:15:07.629649+0000","last_undegraded":"2024-04-23T18:15:07.629649+0000","last_fullsized":"2024-04-23T18:15:07.629649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628988+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:15:07.628988+0000","last_peered":"2024-04-23T18:15:07.628988+0000","last_clean":"2024-04-23T18:15:07.628988+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:15:07.628988+0000","last_undegraded":"2024-04-23T18:15:07.628988+0000","last_fullsized":"2024-04-23T18:15:07.628988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629850+0000","last_change":"2024-04-23T18:15:06.451102+0000","last_active":"2024-04-23T18:15:07.629850+0000","last_peered":"2024-04-23T18:15:07.629850+0000","last_clean":"2024-04-23T18:15:07.629850+0000","last_became_active":"2024-04-23T18:15:06.450776+0000","last_became_peered":"2024-04-23T18:15:06.450776+0000","last_unstale":"2024-04-23T18:15:07.629850+0000","last_undegraded":"2024-04-23T18:15:07.629850+0000","last_fullsized":"2024-04-23T18:15:07.629850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:16:50.619819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429828+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:15:07.429828+0000","last_peered":"2024-04-23T18:15:07.429828+0000","last_clean":"2024-04-23T18:15:07.429828+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:15:07.429828+0000","last_undegraded":"2024-04-23T18:15:07.429828+0000","last_fullsized":"2024-04-23T18:15:07.429828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.709500+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:15:10.709500+0000","last_peered":"2024-04-23T18:15:10.709500+0000","last_clean":"2024-04-23T18:15:10.709500+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:15:10.709500+0000","last_undegraded":"2024-04-23T18:15:10.709500+0000","last_fullsized":"2024-04-23T18:15:10.709500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429550+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:15:07.429550+0000","last_peered":"2024-04-23T18:15:07.429550+0000","last_clean":"2024-04-23T18:15:07.429550+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:15:07.429550+0000","last_undegraded":"2024-04-23T18:15:07.429550+0000","last_fullsized":"2024-04-23T18:15:07.429550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430267+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:15:07.430267+0000","last_peered":"2024-04-23T18:15:07.430267+0000","last_clean":"2024-04-23T18:15:07.430267+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:15:07.430267+0000","last_undegraded":"2024-04-23T18:15:07.430267+0000","last_fullsized":"2024-04-23T18:15:07.430267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628532+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:15:07.628532+0000","last_peered":"2024-04-23T18:15:07.628532+0000","last_clean":"2024-04-23T18:15:07.628532+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:15:07.628532+0000","last_undegraded":"2024-04-23T18:15:07.628532+0000","last_fullsized":"2024-04-23T18:15:07.628532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460090+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:15:07.460090+0000","last_peered":"2024-04-23T18:15:07.460090+0000","last_clean":"2024-04-23T18:15:07.460090+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:15:07.460090+0000","last_undegraded":"2024-04-23T18:15:07.460090+0000","last_fullsized":"2024-04-23T18:15:07.460090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.708256+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:15:10.708256+0000","last_peered":"2024-04-23T18:15:10.708256+0000","last_clean":"2024-04-23T18:15:10.708256+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:15:10.708256+0000","last_undegraded":"2024-04-23T18:15:10.708256+0000","last_fullsized":"2024-04-23T18:15:10.708256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628902+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:15:07.628902+0000","last_peered":"2024-04-23T18:15:07.628902+0000","last_clean":"2024-04-23T18:15:07.628902+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:15:07.628902+0000","last_undegraded":"2024-04-23T18:15:07.628902+0000","last_fullsized":"2024-04-23T18:15:07.628902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460284+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:15:07.460284+0000","last_peered":"2024-04-23T18:15:07.460284+0000","last_clean":"2024-04-23T18:15:07.460284+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:15:07.460284+0000","last_undegraded":"2024-04-23T18:15:07.460284+0000","last_fullsized":"2024-04-23T18:15:07.460284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:10.709290+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:15:10.709290+0000","last_peered":"2024-04-23T18:15:10.709290+0000","last_clean":"2024-04-23T18:15:10.709290+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:15:10.709290+0000","last_undegraded":"2024-04-23T18:15:10.709290+0000","last_fullsized":"2024-04-23T18:15:10.709290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429290+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:15:07.429290+0000","last_peered":"2024-04-23T18:15:07.429290+0000","last_clean":"2024-04-23T18:15:07.429290+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:15:07.429290+0000","last_undegraded":"2024-04-23T18:15:07.429290+0000","last_fullsized":"2024-04-23T18:15:07.429290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.430260+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:15:07.430260+0000","last_peered":"2024-04-23T18:15:07.430260+0000","last_clean":"2024-04-23T18:15:07.430260+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:15:07.430260+0000","last_undegraded":"2024-04-23T18:15:07.430260+0000","last_fullsized":"2024-04-23T18:15:07.430260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628373+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:15:07.628373+0000","last_peered":"2024-04-23T18:15:07.628373+0000","last_clean":"2024-04-23T18:15:07.628373+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:15:07.628373+0000","last_undegraded":"2024-04-23T18:15:07.628373+0000","last_fullsized":"2024-04-23T18:15:07.628373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703335+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:15:09.703335+0000","last_peered":"2024-04-23T18:15:09.703335+0000","last_clean":"2024-04-23T18:15:09.703335+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:15:09.703335+0000","last_undegraded":"2024-04-23T18:15:09.703335+0000","last_fullsized":"2024-04-23T18:15:09.703335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628281+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:15:07.628281+0000","last_peered":"2024-04-23T18:15:07.628281+0000","last_clean":"2024-04-23T18:15:07.628281+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:15:07.628281+0000","last_undegraded":"2024-04-23T18:15:07.628281+0000","last_fullsized":"2024-04-23T18:15:07.628281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.702849+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:15:09.702849+0000","last_peered":"2024-04-23T18:15:09.702849+0000","last_clean":"2024-04-23T18:15:09.702849+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:15:09.702849+0000","last_undegraded":"2024-04-23T18:15:09.702849+0000","last_fullsized":"2024-04-23T18:15:09.702849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.712853+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:15:07.712853+0000","last_peered":"2024-04-23T18:15:07.712853+0000","last_clean":"2024-04-23T18:15:07.712853+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:15:07.712853+0000","last_undegraded":"2024-04-23T18:15:07.712853+0000","last_fullsized":"2024-04-23T18:15:07.712853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628417+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:15:07.628417+0000","last_peered":"2024-04-23T18:15:07.628417+0000","last_clean":"2024-04-23T18:15:07.628417+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:15:07.628417+0000","last_undegraded":"2024-04-23T18:15:07.628417+0000","last_fullsized":"2024-04-23T18:15:07.628417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429250+0000","last_change":"2024-04-23T18:15:06.445013+0000","last_active":"2024-04-23T18:15:07.429250+0000","last_peered":"2024-04-23T18:15:07.429250+0000","last_clean":"2024-04-23T18:15:07.429250+0000","last_became_active":"2024-04-23T18:15:06.443275+0000","last_became_peered":"2024-04-23T18:15:06.443275+0000","last_unstale":"2024-04-23T18:15:07.429250+0000","last_undegraded":"2024-04-23T18:15:07.429250+0000","last_fullsized":"2024-04-23T18:15:07.429250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:58.688700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:09.703111+0000","last_change":"2024-04-23T18:15:05.695339+0000","last_active":"2024-04-23T18:15:09.703111+0000","last_peered":"2024-04-23T18:15:09.703111+0000","last_clean":"2024-04-23T18:15:09.703111+0000","last_became_active":"2024-04-23T18:15:05.695025+0000","last_became_peered":"2024-04-23T18:15:05.695025+0000","last_unstale":"2024-04-23T18:15:09.703111+0000","last_undegraded":"2024-04-23T18:15:09.703111+0000","last_fullsized":"2024-04-23T18:15:09.703111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:55.134576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629123+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:15:07.629123+0000","last_peered":"2024-04-23T18:15:07.629123+0000","last_clean":"2024-04-23T18:15:07.629123+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:15:07.629123+0000","last_undegraded":"2024-04-23T18:15:07.629123+0000","last_fullsized":"2024-04-23T18:15:07.629123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629020+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:15:07.629020+0000","last_peered":"2024-04-23T18:15:07.629020+0000","last_clean":"2024-04-23T18:15:07.629020+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:15:07.629020+0000","last_undegraded":"2024-04-23T18:15:07.629020+0000","last_fullsized":"2024-04-23T18:15:07.629020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629327+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:15:07.629327+0000","last_peered":"2024-04-23T18:15:07.629327+0000","last_clean":"2024-04-23T18:15:07.629327+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:15:07.629327+0000","last_undegraded":"2024-04-23T18:15:07.629327+0000","last_fullsized":"2024-04-23T18:15:07.629327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.773364+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:15:07.773364+0000","last_peered":"2024-04-23T18:15:07.773364+0000","last_clean":"2024-04-23T18:15:07.773364+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:15:07.773364+0000","last_undegraded":"2024-04-23T18:15:07.773364+0000","last_fullsized":"2024-04-23T18:15:07.773364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628534+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:15:07.628534+0000","last_peered":"2024-04-23T18:15:07.628534+0000","last_clean":"2024-04-23T18:15:07.628534+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:15:07.628534+0000","last_undegraded":"2024-04-23T18:15:07.628534+0000","last_fullsized":"2024-04-23T18:15:07.628534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460341+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:15:07.460341+0000","last_peered":"2024-04-23T18:15:07.460341+0000","last_clean":"2024-04-23T18:15:07.460341+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:15:07.460341+0000","last_undegraded":"2024-04-23T18:15:07.460341+0000","last_fullsized":"2024-04-23T18:15:07.460341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629144+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:15:07.629144+0000","last_peered":"2024-04-23T18:15:07.629144+0000","last_clean":"2024-04-23T18:15:07.629144+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:15:07.629144+0000","last_undegraded":"2024-04-23T18:15:07.629144+0000","last_fullsized":"2024-04-23T18:15:07.629144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713038+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:15:07.713038+0000","last_peered":"2024-04-23T18:15:07.713038+0000","last_clean":"2024-04-23T18:15:07.713038+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:15:07.713038+0000","last_undegraded":"2024-04-23T18:15:07.713038+0000","last_fullsized":"2024-04-23T18:15:07.713038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628554+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:15:07.628554+0000","last_peered":"2024-04-23T18:15:07.628554+0000","last_clean":"2024-04-23T18:15:07.628554+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:15:07.628554+0000","last_undegraded":"2024-04-23T18:15:07.628554+0000","last_fullsized":"2024-04-23T18:15:07.628554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628476+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:15:07.628476+0000","last_peered":"2024-04-23T18:15:07.628476+0000","last_clean":"2024-04-23T18:15:07.628476+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:15:07.628476+0000","last_undegraded":"2024-04-23T18:15:07.628476+0000","last_fullsized":"2024-04-23T18:15:07.628476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628089+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:15:07.628089+0000","last_peered":"2024-04-23T18:15:07.628089+0000","last_clean":"2024-04-23T18:15:07.628089+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:15:07.628089+0000","last_undegraded":"2024-04-23T18:15:07.628089+0000","last_fullsized":"2024-04-23T18:15:07.628089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460310+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:15:07.460310+0000","last_peered":"2024-04-23T18:15:07.460310+0000","last_clean":"2024-04-23T18:15:07.460310+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:15:07.460310+0000","last_undegraded":"2024-04-23T18:15:07.460310+0000","last_fullsized":"2024-04-23T18:15:07.460310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713058+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:15:07.713058+0000","last_peered":"2024-04-23T18:15:07.713058+0000","last_clean":"2024-04-23T18:15:07.713058+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:15:07.713058+0000","last_undegraded":"2024-04-23T18:15:07.713058+0000","last_fullsized":"2024-04-23T18:15:07.713058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.629163+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:15:07.629163+0000","last_peered":"2024-04-23T18:15:07.629163+0000","last_clean":"2024-04-23T18:15:07.629163+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:15:07.629163+0000","last_undegraded":"2024-04-23T18:15:07.629163+0000","last_fullsized":"2024-04-23T18:15:07.629163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429132+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:15:07.429132+0000","last_peered":"2024-04-23T18:15:07.429132+0000","last_clean":"2024-04-23T18:15:07.429132+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:15:07.429132+0000","last_undegraded":"2024-04-23T18:15:07.429132+0000","last_fullsized":"2024-04-23T18:15:07.429132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.429841+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:15:07.429841+0000","last_peered":"2024-04-23T18:15:07.429841+0000","last_clean":"2024-04-23T18:15:07.429841+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:15:07.429841+0000","last_undegraded":"2024-04-23T18:15:07.429841+0000","last_fullsized":"2024-04-23T18:15:07.429841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.428961+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:15:07.428961+0000","last_peered":"2024-04-23T18:15:07.428961+0000","last_clean":"2024-04-23T18:15:07.428961+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:15:07.428961+0000","last_undegraded":"2024-04-23T18:15:07.428961+0000","last_fullsized":"2024-04-23T18:15:07.428961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.628923+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:15:07.628923+0000","last_peered":"2024-04-23T18:15:07.628923+0000","last_clean":"2024-04-23T18:15:07.628923+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:15:07.628923+0000","last_undegraded":"2024-04-23T18:15:07.628923+0000","last_fullsized":"2024-04-23T18:15:07.628923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.460394+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:15:07.460394+0000","last_peered":"2024-04-23T18:15:07.460394+0000","last_clean":"2024-04-23T18:15:07.460394+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:15:07.460394+0000","last_undegraded":"2024-04-23T18:15:07.460394+0000","last_fullsized":"2024-04-23T18:15:07.460394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:15:07.713242+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:15:07.713242+0000","last_peered":"2024-04-23T18:15:07.713242+0000","last_clean":"2024-04-23T18:15:07.713242+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:15:07.713242+0000","last_undegraded":"2024-04-23T18:15:07.713242+0000","last_fullsized":"2024-04-23T18:15:07.713242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3906,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":100,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":14148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":443015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1035951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":942080,"data_stored":443799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":443619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40996,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704156,"statfs":{"total":95995035648,"available":95953055744,"internally_reserved":0,"allocated":1536000,"data_stored":1035175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":443105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40428,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704724,"statfs":{"total":95995035648,"available":95953637376,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":445283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":443119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40416,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704736,"statfs":{"total":95995035648,"available":95953649664,"internally_reserved":0,"allocated":929792,"data_stored":443037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32688,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712464,"statfs":{"total":95995035648,"available":95961563136,"internally_reserved":0,"allocated":1548288,"data_stored":1034859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:15:14.829 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T18:15:14.830 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T18:15:14.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: pgmap v287: 97 pgs: 97 active+clean; 580 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 714 B/s rd, 2.9 KiB/s wr, 14 op/s 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1922831343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2109756516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: pgmap v287: 97 pgs: 97 active+clean; 580 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 714 B/s rd, 2.9 KiB/s wr, 14 op/s 2024-04-23T18:15:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:15.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-23T18:15:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1922831343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:15.193 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2109756516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:15.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:15 smithi026 ceph-mon[29364]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:15 smithi131 ceph-mon[33041]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:15 smithi110 ceph-mon[32585]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:16.802 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:16 smithi026 ceph-mon[29364]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:16.802 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:16 smithi026 ceph-mon[29364]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-23T18:15:16.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 -- 172.21.15.26:0/1548731006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4103cf0 msgr2=0x7fafc4104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 --2- 172.21.15.26:0/1548731006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4103cf0 0x7fafc4104150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fafb80099c0 tx=0x7fafb8009cd0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 -- 172.21.15.26:0/1548731006 shutdown_connections 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 --2- 172.21.15.26:0/1548731006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4104690 0x7fafc410af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 --2- 172.21.15.26:0/1548731006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4102af0 0x7fafc4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 --2- 172.21.15.26:0/1548731006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4103cf0 0x7fafc4104150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 -- 172.21.15.26:0/1548731006 >> 172.21.15.26:0/1548731006 conn(0x7fafc40fe090 msgr2=0x7fafc41004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 -- 172.21.15.26:0/1548731006 shutdown_connections 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.828+0000 7fafc8862700 1 -- 172.21.15.26:0/1548731006 wait complete. 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 Processor -- start 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 -- start start 2024-04-23T18:15:16.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4102af0 0x7fafc419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4103cf0 0x7fafc419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fafc410e680 con 0x7fafc4104690 2024-04-23T18:15:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fafc410e500 con 0x7fafc4102af0 2024-04-23T18:15:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc8862700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fafc410e800 con 0x7fafc4103cf0 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc2d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc2d9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38280/0 (socket says 172.21.15.26:38280) 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc2d9d700 1 -- 172.21.15.26:0/485362920 learned_addr learned my addr 172.21.15.26:0/485362920 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc259c700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4102af0 0x7fafc419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.829+0000 7fafc1d9b700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4103cf0 0x7fafc419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 -- 172.21.15.26:0/485362920 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4103cf0 msgr2=0x7fafc419b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4103cf0 0x7fafc419b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 -- 172.21.15.26:0/485362920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4102af0 msgr2=0x7fafc419aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4102af0 0x7fafc419aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 -- 172.21.15.26:0/485362920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc41a2b00 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc2d9d700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b9a0 tx=0x7fafbc00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00d850 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafbc00e420 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc41a2df0 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc017640 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.830+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc41a32f0 con 0x7fafc4104690 2024-04-23T18:15:16.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.831+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc4050f80 con 0x7fafc4104690 2024-04-23T18:15:16.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.832+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fafbc00d9f0 con 0x7fafc4104690 2024-04-23T18:15:16.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.833+0000 7fafb37fe700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 0x7fafac0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:16.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.833+0000 7fafc259c700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 0x7fafac0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:16.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.833+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fafbc0d0150 con 0x7fafc4104690 2024-04-23T18:15:16.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.834+0000 7fafc259c700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 0x7fafac0a62c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fafc4103b50 tx=0x7fafb4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:16.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:16.837+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafbc013030 con 0x7fafc4104690 2024-04-23T18:15:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:16 smithi131 ceph-mon[33041]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:16 smithi131 ceph-mon[33041]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-23T18:15:17.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.132+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fafc41a37e0 con 0x7fafc4104690 2024-04-23T18:15:17.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.135+0000 7fafb37fe700 1 -- 172.21.15.26:0/485362920 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fafbc09cb00 con 0x7fafc4104690 2024-04-23T18:15:17.129 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-04-23T18:15:17.129 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC1+idmiJ/8BxAAcDcU+3zmPODAudx9igr27A== 2024-04-23T18:15:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 msgr2=0x7fafac0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 0x7fafac0a62c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fafc4103b50 tx=0x7fafb4009190 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 msgr2=0x7fafc41a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b9a0 tx=0x7fafbc00bcb0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 shutdown_connections 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fafac0a3e00 0x7fafac0a62c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fafc4104690 0x7fafc41a2460 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafc4102af0 0x7fafc419aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 --2- 172.21.15.26:0/485362920 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafc4103cf0 0x7fafc419b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.137+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 >> 172.21.15.26:0/485362920 conn(0x7fafc40fe090 msgr2=0x7fafc4074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.138+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 shutdown_connections 2024-04-23T18:15:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:17.138+0000 7fafc8862700 1 -- 172.21.15.26:0/485362920 wait complete. 2024-04-23T18:15:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:16 smithi110 ceph-mon[32585]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:15:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:16 smithi110 ceph-mon[32585]: pgmap v288: 97 pgs: 97 active+clean; 581 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-23T18:15:17.730 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:15:17.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T18:15:17.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T18:15:17.754 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:15:17.754 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:15:17.770 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T18:15:17.770 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T18:15:17.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T18:15:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:15:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:15:18.058 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:18.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:15:18.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/485362920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:15:18.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:18 smithi026 ceph-mon[29364]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-04-23T18:15:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:18 smithi131 ceph-mon[33041]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-04-23T18:15:19.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:18 smithi110 ceph-mon[32585]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-04-23T18:15:19.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 -- 172.21.15.26:0/4148519171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0108a50 msgr2=0x7f7da0108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:19.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 --2- 172.21.15.26:0/4148519171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0108a50 0x7f7da0108e30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009990 tx=0x7f7d9c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 -- 172.21.15.26:0/4148519171 shutdown_connections 2024-04-23T18:15:19.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 --2- 172.21.15.26:0/4148519171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0103450 0x7f7da010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 --2- 172.21.15.26:0/4148519171 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0108a50 0x7f7da0108e30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 --2- 172.21.15.26:0/4148519171 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0102a90 0x7f7da0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 -- 172.21.15.26:0/4148519171 >> 172.21.15.26:0/4148519171 conn(0x7f7da0079ba0 msgr2=0x7f7da0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 -- 172.21.15.26:0/4148519171 shutdown_connections 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.854+0000 7f7da87a8700 1 -- 172.21.15.26:0/4148519171 wait complete. 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 Processor -- start 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 -- start start 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0102a90 0x7f7da0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0108a50 0x7f7da019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0112c00 con 0x7f7da0103450 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0112a80 con 0x7f7da0108a50 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da87a8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0112d80 con 0x7f7da0102a90 2024-04-23T18:15:19.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da5d43700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da5d43700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57324/0 (socket says 172.21.15.26:57324) 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da5d43700 1 -- 172.21.15.26:0/3321061753 learned_addr learned my addr 172.21.15.26:0/3321061753 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.855+0000 7f7da6d45700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0108a50 0x7f7da019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 -- 172.21.15.26:0/3321061753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0102a90 msgr2=0x7f7da0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0102a90 0x7f7da0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 -- 172.21.15.26:0/3321061753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0108a50 msgr2=0x7f7da019d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0108a50 0x7f7da019d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 -- 172.21.15.26:0/3321061753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da019e120 con 0x7f7da0103450 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da5d43700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000d980 tx=0x7f7d9000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90017070 con 0x7f7da0103450 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d90005020 con 0x7f7da0103450 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da019e410 con 0x7f7da0103450 2024-04-23T18:15:19.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90009b50 con 0x7f7da0103450 2024-04-23T18:15:19.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.856+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da01a5c90 con 0x7f7da0103450 2024-04-23T18:15:19.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.857+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da0069730 con 0x7f7da0103450 2024-04-23T18:15:19.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.858+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7d900051d0 con 0x7f7da0103450 2024-04-23T18:15:19.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.859+0000 7f7d977fe700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 0x7f7d8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:19.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.859+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7d900d0320 con 0x7f7da0103450 2024-04-23T18:15:19.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.859+0000 7f7da6544700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 0x7f7d8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:19.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.859+0000 7f7da6544700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 0x7f7d8c0a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0098e0 tx=0x7f7d9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:19.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:19.863+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d90020020 con 0x7f7da0103450 2024-04-23T18:15:20.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.131+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7da002df80 con 0x7f7da0103450 2024-04-23T18:15:20.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.136+0000 7f7d977fe700 1 -- 172.21.15.26:0/3321061753 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264993 (secure 0 0 0) 0x7f7d9009cd50 con 0x7f7da0103450 2024-04-23T18:15:20.133 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:20.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 msgr2=0x7f7d8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:20.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 0x7f7d8c0a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0098e0 tx=0x7f7d9c005fb0 comp rx=0 tx=0).stop 2024-04-23T18:15:20.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 msgr2=0x7f7da0199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000d980 tx=0x7f7d9000dc90 comp rx=0 tx=0).stop 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 shutdown_connections 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f7d8c0a3cd0 0x7f7d8c0a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0108a50 0x7f7da019d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0102a90 0x7f7da0199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 --2- 172.21.15.26:0/3321061753 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0103450 0x7f7da0199650 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 >> 172.21.15.26:0/3321061753 conn(0x7f7da0079ba0 msgr2=0x7f7da0101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 shutdown_connections 2024-04-23T18:15:20.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:20.139+0000 7f7da87a8700 1 -- 172.21.15.26:0/3321061753 wait complete. 2024-04-23T18:15:20.832 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":270207440},{"type":"v1","addr":"172.21.15.26:6801","nonce":270207440}]},"active_addr":"172.21.15.26:6801/270207440","active_change":"2024-04-23T18:08:10.758102+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14223,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.26:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":736937134}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":497279081}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1107719668}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2404051623}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1760552523}]}]} 2024-04-23T18:15:20.838 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T18:15:20.838 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T18:15:20.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:15:20.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:20 smithi131 ceph-mon[33041]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 17 op/s 2024-04-23T18:15:20.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:20 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3321061753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:15:21.088 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:21.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:20 smithi110 ceph-mon[32585]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 17 op/s 2024-04-23T18:15:21.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:20 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3321061753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:15:21.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:20 smithi026 ceph-mon[29364]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 17 op/s 2024-04-23T18:15:21.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:20 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3321061753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:15:22.797 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:22 smithi026 ceph-mon[29364]: pgmap v291: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-04-23T18:15:22.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 -- 172.21.15.26:0/335269973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94101940 msgr2=0x7fdb94101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 --2- 172.21.15.26:0/335269973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94101940 0x7fdb94101dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009990 tx=0x7fdb7c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 -- 172.21.15.26:0/335269973 shutdown_connections 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 --2- 172.21.15.26:0/335269973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94102300 0x7fdb9410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 --2- 172.21.15.26:0/335269973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94107990 0x7fdb94107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 --2- 172.21.15.26:0/335269973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94101940 0x7fdb94101dc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 -- 172.21.15.26:0/335269973 >> 172.21.15.26:0/335269973 conn(0x7fdb940788f0 msgr2=0x7fdb94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 -- 172.21.15.26:0/335269973 shutdown_connections 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.808+0000 7fdb9a192700 1 -- 172.21.15.26:0/335269973 wait complete. 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 Processor -- start 2024-04-23T18:15:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 -- start start 2024-04-23T18:15:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94101940 0x7fdb94197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94102300 0x7fdb94198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94111af0 con 0x7fdb94107990 2024-04-23T18:15:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94111970 con 0x7fdb94102300 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb9a192700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94111c70 con 0x7fdb94101940 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb93fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.809+0000 7fdb937fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94101940 0x7fdb94197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57330/0 (socket says 172.21.15.26:57330) 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 -- 172.21.15.26:0/2990393714 learned_addr learned my addr 172.21.15.26:0/2990393714 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb92ffd700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94102300 0x7fdb94198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 -- 172.21.15.26:0/2990393714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94101940 msgr2=0x7fdb94197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94101940 0x7fdb94197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 -- 172.21.15.26:0/2990393714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94102300 msgr2=0x7fdb94198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94102300 0x7fdb94198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 -- 172.21.15.26:0/2990393714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9419cfd0 con 0x7fdb94107990 2024-04-23T18:15:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb93fff700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400d980 tx=0x7fdb8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84017070 con 0x7fdb94107990 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9419d2c0 con 0x7fdb94107990 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb84005020 con 0x7fdb94107990 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8400fa10 con 0x7fdb94107990 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.810+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb941a4b40 con 0x7fdb94107990 2024-04-23T18:15:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.811+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb94069730 con 0x7fdb94107990 2024-04-23T18:15:22.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.812+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdb84004a30 con 0x7fdb94107990 2024-04-23T18:15:22.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.813+0000 7fdb90ff9700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 0x7fdb800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:22.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.813+0000 7fdb937fe700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 0x7fdb800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:22.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.813+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdb840d0280 con 0x7fdb94107990 2024-04-23T18:15:22.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.813+0000 7fdb937fe700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 0x7fdb800a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009550 tx=0x7fdb8c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:22.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:22.817+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb84020020 con 0x7fdb94107990 2024-04-23T18:15:22.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:22 smithi131 ceph-mon[33041]: pgmap v291: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-04-23T18:15:23.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.024+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb9402df80 con 0x7fdb94107990 2024-04-23T18:15:23.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.026+0000 7fdb90ff9700 1 -- 172.21.15.26:0/2990393714 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7fdb8409cbb0 con 0x7fdb94107990 2024-04-23T18:15:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:23.020 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":86,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:15:07.425796+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 msgr2=0x7fdb800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 0x7fdb800a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009550 tx=0x7fdb8c0091b0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 msgr2=0x7fdb9419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400d980 tx=0x7fdb8400dc90 comp rx=0 tx=0).stop 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 shutdown_connections 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdb800a3cd0 0x7fdb800a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdb94107990 0x7fdb9419c890 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb94101940 0x7fdb94197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 --2- 172.21.15.26:0/2990393714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb94102300 0x7fdb94198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 >> 172.21.15.26:0/2990393714 conn(0x7fdb940788f0 msgr2=0x7fdb940ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:23.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 shutdown_connections 2024-04-23T18:15:23.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:23.028+0000 7fdb9a192700 1 -- 172.21.15.26:0/2990393714 wait complete. 2024-04-23T18:15:23.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:22 smithi110 ceph-mon[32585]: pgmap v291: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-04-23T18:15:23.687 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T18:15:23.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:15:23.938 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:23.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:23 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2990393714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:23.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:23 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2990393714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:24.093 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:23 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2990393714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:24 smithi026 ceph-mon[29364]: pgmap v292: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2024-04-23T18:15:24.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:24 smithi131 ceph-mon[33041]: pgmap v292: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2024-04-23T18:15:25.113 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:24 smithi110 ceph-mon[32585]: pgmap v292: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2024-04-23T18:15:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 -- 172.21.15.26:0/317389505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8103480 msgr2=0x7f25c810f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/317389505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8103480 0x7f25c810f980 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009990 tx=0x7f25b8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 -- 172.21.15.26:0/317389505 shutdown_connections 2024-04-23T18:15:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/317389505 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8103480 0x7f25c810f980 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/317389505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 0x7f25c8108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/317389505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8102ac0 0x7f25c8102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 -- 172.21.15.26:0/317389505 >> 172.21.15.26:0/317389505 conn(0x7f25c8079bf0 msgr2=0x7f25c807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:25.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 -- 172.21.15.26:0/317389505 shutdown_connections 2024-04-23T18:15:25.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.607+0000 7f25cfbe7700 1 -- 172.21.15.26:0/317389505 wait complete. 2024-04-23T18:15:25.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 Processor -- start 2024-04-23T18:15:25.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 -- start start 2024-04-23T18:15:25.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:25.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8103480 0x7f25c8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 0x7f25c80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f25c8112cf0 con 0x7f25c8102ac0 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25c8112b70 con 0x7f25c8108ab0 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.608+0000 7f25cfbe7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f25c8112e70 con 0x7f25c8103480 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57348/0 (socket says 172.21.15.26:57348) 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 -- 172.21.15.26:0/1450918269 learned_addr learned my addr 172.21.15.26:0/1450918269 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25ce184700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 0x7f25c80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd182700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8103480 0x7f25c8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 -- 172.21.15.26:0/1450918269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8103480 msgr2=0x7f25c8075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8103480 0x7f25c8075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 -- 172.21.15.26:0/1450918269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 msgr2=0x7f25c80757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 0x7f25c80757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 -- 172.21.15.26:0/1450918269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c81a7690 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cd983700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f25c0007f10 tx=0x7f25c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004db0 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c0003a00 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c00045e0 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.609+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c81a7980 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.610+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c81a7ec0 con 0x7f25c8102ac0 2024-04-23T18:15:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.611+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c8069730 con 0x7f25c8102ac0 2024-04-23T18:15:25.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.613+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f25c001c020 con 0x7f25c8102ac0 2024-04-23T18:15:25.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.613+0000 7f25c6ffd700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 0x7f25b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:25.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.613+0000 7f25cd182700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 0x7f25b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:25.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.613+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f25c00cfc20 con 0x7f25c8102ac0 2024-04-23T18:15:25.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.614+0000 7f25cd182700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 0x7f25b00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f25c807a080 tx=0x7f25bc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:25.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.618+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25c0010070 con 0x7f25c8102ac0 2024-04-23T18:15:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.827+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25c81a8090 con 0x7f25c8102ac0 2024-04-23T18:15:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.829+0000 7f25c6ffd700 1 -- 172.21.15.26:0/1450918269 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7f25c009c650 con 0x7f25c8102ac0 2024-04-23T18:15:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:15:25.823 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":86,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:15:07.425796+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 msgr2=0x7f25b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 0x7f25b00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f25c807a080 tx=0x7f25bc009190 comp rx=0 tx=0).stop 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 msgr2=0x7f25c8074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f25c0007f10 tx=0x7f25c000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 shutdown_connections 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f25b00a3cd0 0x7f25b00a6190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c8108ab0 0x7f25c80757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25c8102ac0 0x7f25c8074d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 --2- 172.21.15.26:0/1450918269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25c8103480 0x7f25c8075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 >> 172.21.15.26:0/1450918269 conn(0x7f25c8079bf0 msgr2=0x7f25c8100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 shutdown_connections 2024-04-23T18:15:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:25.831+0000 7f25cfbe7700 1 -- 172.21.15.26:0/1450918269 wait complete. 2024-04-23T18:15:26.466 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T18:15:26.466 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T18:15:26.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T18:15:26.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T18:15:26.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T18:15:26.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T18:15:26.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T18:15:26.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T18:15:26.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T18:15:26.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T18:15:26.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T18:15:26.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T18:15:26.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:26 smithi131 ceph-mon[33041]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 KiB/s wr, 11 op/s 2024-04-23T18:15:26.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:26 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1450918269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:27.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:26 smithi110 ceph-mon[32585]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 KiB/s wr, 11 op/s 2024-04-23T18:15:27.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:26 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1450918269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:27.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:26 smithi026 ceph-mon[29364]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 KiB/s wr, 11 op/s 2024-04-23T18:15:27.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:26 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1450918269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:27.214 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:27.643 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:27.644 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:28.110 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:28.112 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:28.218 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:28.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:28 smithi026 ceph-mon[29364]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-04-23T18:15:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:28 smithi131 ceph-mon[33041]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-04-23T18:15:29.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:28 smithi110 ceph-mon[32585]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.149+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/2669319669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec074c70 msgr2=0x7fcdec075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.149+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/2669319669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec074c70 0x7fcdec075050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009990 tx=0x7fcde4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.149+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/2669319669 shutdown_connections 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/2669319669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdec10f470 0x7fcdec111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/2669319669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec074c70 0x7fcdec075050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/2669319669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdec075590 0x7fcdec10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/2669319669 >> 172.21.15.26:0/2669319669 conn(0x7fcdec079bf0 msgr2=0x7fcdec07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/2669319669 shutdown_connections 2024-04-23T18:15:30.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/2669319669 wait complete. 2024-04-23T18:15:30.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.150+0000 7fcdf1e8a700 1 Processor -- start 2024-04-23T18:15:30.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 -- start start 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdec074c70 0x7fcdec19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 0x7fcdec19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdec10f470 0x7fcdec19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec114ec0 con 0x7fcdec075590 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec114d40 con 0x7fcdec074c70 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdf1e8a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec115040 con 0x7fcdec10f470 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 0x7fcdec19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 0x7fcdec19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35058/0 (socket says 172.21.15.26:35058) 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 -- 172.21.15.26:0/5696245 learned_addr learned my addr 172.21.15.26:0/5696245 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdebfff700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdec10f470 0x7fcdec19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeb7fe700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdec074c70 0x7fcdec19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 -- 172.21.15.26:0/5696245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdec10f470 msgr2=0x7fcdec19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdec10f470 0x7fcdec19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 -- 172.21.15.26:0/5696245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdec074c70 msgr2=0x7fcdec19b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdec074c70 0x7fcdec19b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.151+0000 7fcdeaffd700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1a0390 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcdeaffd700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 0x7fcdec19b910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d950 tx=0x7fcdd400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4016070 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec1a0680 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4009df0 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd400f3c0 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.152+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec1a7f00 con 0x7fcdec075590 2024-04-23T18:15:30.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.153+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcdec0630d0 con 0x7fcdec075590 2024-04-23T18:15:30.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.154+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcdd4019070 con 0x7fcdec075590 2024-04-23T18:15:30.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.154+0000 7fcde8ff9700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcdd80a3cd0 0x7fcdd80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:30.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcdeb7fe700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcdd80a3cd0 0x7fcdd80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:30.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fcdd40d0500 con 0x7fcdec075590 2024-04-23T18:15:30.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcde8ff9700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] conn(0x7fcdd80ad4b0 0x7fcdd80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdd4013ec0 con 0x7fcdd80ad4b0 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fcdd40d09e0 con 0x7fcdec075590 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcdebfff700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] conn(0x7fcdd80ad4b0 0x7fcdd80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcdeb7fe700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcdd80a3cd0 0x7fcdd80a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcde400b5c0 tx=0x7fcde401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.155+0000 7fcdebfff700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] conn(0x7fcdd80ad4b0 0x7fcdd80af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:30.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.156+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== osd.0 v2:172.21.15.26:6802/75846197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdd4013ec0 con 0x7fcdd80ad4b0 2024-04-23T18:15:30.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.172+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 --> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcdec063580 con 0x7fcdd80ad4b0 2024-04-23T18:15:30.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcde8ff9700 1 -- 172.21.15.26:0/5696245 <== osd.0 v2:172.21.15.26:6802/75846197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcdec063580 con 0x7fcdd80ad4b0 2024-04-23T18:15:30.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] conn(0x7fcdd80ad4b0 msgr2=0x7fcdd80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] conn(0x7fcdd80ad4b0 0x7fcdd80af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:30.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcdd80a3cd0 msgr2=0x7fcdd80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcdd80a3cd0 0x7fcdd80a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcde400b5c0 tx=0x7fcde401a040 comp rx=0 tx=0).stop 2024-04-23T18:15:30.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 msgr2=0x7fcdec19b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 --2- 172.21.15.26:0/5696245 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcdec075590 0x7fcdec19b910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d950 tx=0x7fcdd400dc60 comp rx=0 tx=0).stop 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdebfff700 1 -- 172.21.15.26:0/5696245 reap_dead start 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 shutdown_connections 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 >> 172.21.15.26:0/5696245 conn(0x7fcdec079bf0 msgr2=0x7fcdec0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 shutdown_connections 2024-04-23T18:15:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:30.173+0000 7fcdf1e8a700 1 -- 172.21.15.26:0/5696245 wait complete. 2024-04-23T18:15:30.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:30 smithi131 ceph-mon[33041]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:15:31.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:30 smithi110 ceph-mon[32585]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:15:31.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:30 smithi026 ceph-mon[29364]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:15:32.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:32 smithi026 ceph-mon[29364]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-23T18:15:32.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:32 smithi131 ceph-mon[33041]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-23T18:15:33.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:32 smithi110 ceph-mon[32585]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-23T18:15:33.763 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:34.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:34 smithi026 ceph-mon[29364]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:34 smithi131 ceph-mon[33041]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:35.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:34 smithi110 ceph-mon[32585]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:35.827 INFO:teuthology.orchestra.run.smithi026.stdout:34359738440 2024-04-23T18:15:35.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T18:15:36.193 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:36.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:36 smithi026 ceph-mon[29364]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:15:37.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:36 smithi110 ceph-mon[32585]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:15:37.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:36 smithi131 ceph-mon[33041]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:15:37.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/1763472195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 msgr2=0x7f4fd8101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/1763472195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8101dc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009990 tx=0x7f4fcc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/1763472195 shutdown_connections 2024-04-23T18:15:37.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/1763472195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8102300 0x7f4fd810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/1763472195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fd8107990 0x7f4fd8107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/1763472195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8101dc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/1763472195 >> 172.21.15.26:0/1763472195 conn(0x7f4fd80788f0 msgr2=0x7f4fd8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/1763472195 shutdown_connections 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.795+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/1763472195 wait complete. 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 Processor -- start 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 -- start start 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fd8102300 0x7f4fd8198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8107990 0x7f4fd81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8111b70 con 0x7f4fd8101940 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81119f0 con 0x7f4fd8102300 2024-04-23T18:15:37.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.796+0000 7f4fdf8fe700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8111cf0 con 0x7f4fd8107990 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:35070/0 (socket says 172.21.15.26:35070) 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 -- 172.21.15.26:0/3759981094 learned_addr learned my addr 172.21.15.26:0/3759981094 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdce99700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fd8102300 0x7f4fd8198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdde9b700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8107990 0x7f4fd81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 -- 172.21.15.26:0/3759981094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8107990 msgr2=0x7f4fd81988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8107990 0x7f4fd81988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 -- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fd8102300 msgr2=0x7f4fd8198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fd8102300 0x7f4fd8198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd81a7630 con 0x7f4fd8101940 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdde9b700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4fd8107990 0x7f4fd81988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdd69a700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8197e20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000ba50 tx=0x7f4fd000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0019070 con 0x7f4fd8101940 2024-04-23T18:15:37.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0014440 con 0x7f4fd8101940 2024-04-23T18:15:37.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd81a7920 con 0x7f4fd8101940 2024-04-23T18:15:37.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.797+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000ce60 con 0x7f4fd8101940 2024-04-23T18:15:37.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.798+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd81a7e80 con 0x7f4fd8101940 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.798+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fd8063150 con 0x7f4fd8101940 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.800+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4fd001c070 con 0x7f4fd8101940 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.800+0000 7f4fd67fc700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4fc00a3d50 0x7f4fc00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.800+0000 7f4fdce99700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4fc00a3d50 0x7f4fc00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.800+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4fd00cfde0 con 0x7f4fd8101940 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.801+0000 7f4fd67fc700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] conn(0x7f4fc00ad530 0x7f4fc00af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.801+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fd0015e80 con 0x7f4fc00ad530 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.801+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f4fd00d02c0 con 0x7f4fd8101940 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.801+0000 7f4fdce99700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4fc00a3d50 0x7f4fc00a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0098e0 tx=0x7f4fcc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.801+0000 7f4fdde9b700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] conn(0x7f4fc00ad530 0x7f4fc00af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:37.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.802+0000 7f4fdde9b700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] conn(0x7f4fc00ad530 0x7f4fc00af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:37.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.802+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== osd.1 v2:172.21.15.110:6800/1354854884 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fd0015e80 con 0x7f4fc00ad530 2024-04-23T18:15:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.820+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 --> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4fd80635d0 con 0x7f4fc00ad530 2024-04-23T18:15:37.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fd67fc700 1 -- 172.21.15.26:0/3759981094 <== osd.1 v2:172.21.15.110:6800/1354854884 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4fd80635d0 con 0x7f4fc00ad530 2024-04-23T18:15:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] conn(0x7f4fc00ad530 msgr2=0x7f4fc00af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] conn(0x7f4fc00ad530 0x7f4fc00af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4fc00a3d50 msgr2=0x7f4fc00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4fc00a3d50 0x7f4fc00a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0098e0 tx=0x7f4fcc005960 comp rx=0 tx=0).stop 2024-04-23T18:15:37.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 msgr2=0x7f4fd8197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 --2- 172.21.15.26:0/3759981094 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4fd8101940 0x7f4fd8197e20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000ba50 tx=0x7f4fd000bd60 comp rx=0 tx=0).stop 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdde9b700 1 -- 172.21.15.26:0/3759981094 reap_dead start 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.821+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 shutdown_connections 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.822+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 >> 172.21.15.26:0/3759981094 conn(0x7f4fd80788f0 msgr2=0x7f4fd80ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.822+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 shutdown_connections 2024-04-23T18:15:37.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:37.822+0000 7f4fdf8fe700 1 -- 172.21.15.26:0/3759981094 wait complete. 2024-04-23T18:15:38.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:38 smithi026 ceph-mon[29364]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 -- 172.21.15.26:0/4035254194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 msgr2=0x7fa5fc102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 --2- 172.21.15.26:0/4035254194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc102f70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009990 tx=0x7fa5f0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 -- 172.21.15.26:0/4035254194 shutdown_connections 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 --2- 172.21.15.26:0/4035254194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5fc1034b0 0x7fa5fc10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 --2- 172.21.15.26:0/4035254194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5fc108a80 0x7fa5fc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 --2- 172.21.15.26:0/4035254194 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc102f70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 -- 172.21.15.26:0/4035254194 >> 172.21.15.26:0/4035254194 conn(0x7fa5fc079b20 msgr2=0x7fa5fc079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 -- 172.21.15.26:0/4035254194 shutdown_connections 2024-04-23T18:15:38.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 -- 172.21.15.26:0/4035254194 wait complete. 2024-04-23T18:15:38.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.979+0000 7fa6018c0700 1 Processor -- start 2024-04-23T18:15:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 -- start start 2024-04-23T18:15:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5fc1034b0 0x7fa5fc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5fc108a80 0x7fa5fc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc112c30 con 0x7fa5fc102af0 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc112ab0 con 0x7fa5fc108a80 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa6018c0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc112db0 con 0x7fa5fc1034b0 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34428/0 (socket says 172.21.15.26:34428) 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 -- 172.21.15.26:0/2950229857 learned_addr learned my addr 172.21.15.26:0/2950229857 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5fb7fe700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5fc108a80 0x7fa5fc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5fa7fc700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5fc1034b0 0x7fa5fc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 -- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5fc1034b0 msgr2=0x7fa5fc075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5fc1034b0 0x7fa5fc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 -- 172.21.15.26:0/2950229857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5fc108a80 msgr2=0x7fa5fc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5fc108a80 0x7fa5fc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.980+0000 7fa5faffd700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc1a7690 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa5faffd700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc074d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c970 tx=0x7fa5ec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec007aa0 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc1a7980 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5ec004ce0 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec005510 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.981+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc1a7ec0 con 0x7fa5fc102af0 2024-04-23T18:15:38.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.982+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5fc063110 con 0x7fa5fc102af0 2024-04-23T18:15:38.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.983+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa5ec007c40 con 0x7fa5fc102af0 2024-04-23T18:15:38.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.983+0000 7fa6008be700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5e40a3d50 0x7fa5e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa5fa7fc700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5e40a3d50 0x7fa5e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa5ec0d0610 con 0x7fa5fc102af0 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa6008be700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] conn(0x7fa5e40ad530 0x7fa5e40af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5ec00eec0 con 0x7fa5e40ad530 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fa5ec0d0a00 con 0x7fa5fc102af0 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa5fa7fc700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5e40a3d50 0x7fa5e40a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000b5c0 tx=0x7fa5f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.984+0000 7fa5fb7fe700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] conn(0x7fa5e40ad530 0x7fa5e40af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.985+0000 7fa5fb7fe700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] conn(0x7fa5e40ad530 0x7fa5e40af950 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:38.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:38.986+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== osd.2 v2:172.21.15.131:6800/2462004280 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5ec00eec0 con 0x7fa5e40ad530 2024-04-23T18:15:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.002+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 --> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa5fc063510 con 0x7fa5e40ad530 2024-04-23T18:15:38.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6008be700 1 -- 172.21.15.26:0/2950229857 <== osd.2 v2:172.21.15.131:6800/2462004280 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa5fc063510 con 0x7fa5e40ad530 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] conn(0x7fa5e40ad530 msgr2=0x7fa5e40af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] conn(0x7fa5e40ad530 0x7fa5e40af950 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5e40a3d50 msgr2=0x7fa5e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5e40a3d50 0x7fa5e40a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000b5c0 tx=0x7fa5f0005fb0 comp rx=0 tx=0).stop 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 msgr2=0x7fa5fc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 --2- 172.21.15.26:0/2950229857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa5fc102af0 0x7fa5fc074d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c970 tx=0x7fa5ec00cc80 comp rx=0 tx=0).stop 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa5fb7fe700 1 -- 172.21.15.26:0/2950229857 reap_dead start 2024-04-23T18:15:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 shutdown_connections 2024-04-23T18:15:38.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 >> 172.21.15.26:0/2950229857 conn(0x7fa5fc079b20 msgr2=0x7fa5fc101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:38.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 shutdown_connections 2024-04-23T18:15:38.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.003+0000 7fa6018c0700 1 -- 172.21.15.26:0/2950229857 wait complete. 2024-04-23T18:15:39.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:38 smithi110 ceph-mon[32585]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:39.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:38 smithi131 ceph-mon[33041]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 -- 172.21.15.26:0/2038089714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 msgr2=0x7f0ca406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 --2- 172.21.15.26:0/2038089714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca406b440 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009990 tx=0x7f0c98009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 -- 172.21.15.26:0/2038089714 shutdown_connections 2024-04-23T18:15:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 --2- 172.21.15.26:0/2038089714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 0x7f0ca410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 --2- 172.21.15.26:0/2038089714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ca4108a20 0x7f0ca4108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 --2- 172.21.15.26:0/2038089714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca406b440 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 -- 172.21.15.26:0/2038089714 >> 172.21.15.26:0/2038089714 conn(0x7f0ca40788b0 msgr2=0x7f0ca4078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:39.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 -- 172.21.15.26:0/2038089714 shutdown_connections 2024-04-23T18:15:39.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.815+0000 7f0ca9671700 1 -- 172.21.15.26:0/2038089714 wait complete. 2024-04-23T18:15:39.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 Processor -- start 2024-04-23T18:15:39.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 -- start start 2024-04-23T18:15:39.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca41992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 0x7f0ca41997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ca4108a20 0x7f0ca419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4112ca0 con 0x7f0ca406afe0 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4112b20 con 0x7f0ca4108a20 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca9671700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4112e20 con 0x7f0ca406b980 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca41992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:39.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca27fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 0x7f0ca41997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca41992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34444/0 (socket says 172.21.15.26:34444) 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca2ffd700 1 -- 172.21.15.26:0/1498287712 learned_addr learned my addr 172.21.15.26:0/1498287712 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.816+0000 7f0ca37fe700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ca4108a20 0x7f0ca419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 -- 172.21.15.26:0/1498287712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ca4108a20 msgr2=0x7f0ca419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ca4108a20 0x7f0ca419db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 -- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 msgr2=0x7f0ca41992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0ca406afe0 0x7f0ca41992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca419e2b0 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca27fc700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 0x7f0ca41997e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c98006010 tx=0x7f0c980057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98027070 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca419e540 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c98003fe0 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98022bd0 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.817+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca41a5e20 con 0x7f0ca406b980 2024-04-23T18:15:39.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.818+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ca40631d0 con 0x7f0ca406b980 2024-04-23T18:15:39.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.820+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0c9802a070 con 0x7f0ca406b980 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0c8bfff700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0c8c0a3cd0 0x7f0c8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0ca2ffd700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0c8c0a3cd0 0x7f0c8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0c980d1ea0 con 0x7f0ca406b980 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0c8bfff700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] conn(0x7f0c8c0ad4b0 0x7f0c8c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c98001d30 con 0x7f0c8c0ad4b0 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f0c980d2380 con 0x7f0ca406b980 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0ca2ffd700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0c8c0a3cd0 0x7f0c8c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400e250 tx=0x7f0c9400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.821+0000 7f0ca37fe700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] conn(0x7f0c8c0ad4b0 0x7f0c8c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.822+0000 7f0ca37fe700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] conn(0x7f0c8c0ad4b0 0x7f0c8c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:39.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.822+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== osd.8 v2:172.21.15.131:6816/3490838385 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c98001d30 con 0x7f0c8c0ad4b0 2024-04-23T18:15:39.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.839+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 --> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ca40635d0 con 0x7f0c8c0ad4b0 2024-04-23T18:15:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0c8bfff700 1 -- 172.21.15.26:0/1498287712 <== osd.8 v2:172.21.15.131:6816/3490838385 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0ca40635d0 con 0x7f0c8c0ad4b0 2024-04-23T18:15:39.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] conn(0x7f0c8c0ad4b0 msgr2=0x7f0c8c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] conn(0x7f0c8c0ad4b0 0x7f0c8c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0c8c0a3cd0 msgr2=0x7f0c8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0c8c0a3cd0 0x7f0c8c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400e250 tx=0x7f0c9400b040 comp rx=0 tx=0).stop 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 msgr2=0x7f0ca41997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 --2- 172.21.15.26:0/1498287712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca406b980 0x7f0ca41997e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c98006010 tx=0x7f0c980057c0 comp rx=0 tx=0).stop 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca37fe700 1 -- 172.21.15.26:0/1498287712 reap_dead start 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 shutdown_connections 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 >> 172.21.15.26:0/1498287712 conn(0x7f0ca40788b0 msgr2=0x7f0ca40ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 shutdown_connections 2024-04-23T18:15:39.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:39.840+0000 7f0ca9671700 1 -- 172.21.15.26:0/1498287712 wait complete. 2024-04-23T18:15:40.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:40 smithi026 ceph-mon[29364]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:41.044 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:40 smithi131 ceph-mon[33041]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:41.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:40 smithi110 ceph-mon[32585]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 -- 172.21.15.26:0/1770468611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57501026e0 msgr2=0x7f575010e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 --2- 172.21.15.26:0/1770468611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57501026e0 0x7f575010e770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f574800b0a0 tx=0x7f574800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 -- 172.21.15.26:0/1770468611 shutdown_connections 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 --2- 172.21.15.26:0/1770468611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f57501026e0 0x7f575010e770 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 --2- 172.21.15.26:0/1770468611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5750106990 0x7f5750106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 --2- 172.21.15.26:0/1770468611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750101d20 0x7f57501021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 -- 172.21.15.26:0/1770468611 >> 172.21.15.26:0/1770468611 conn(0x7f57500788f0 msgr2=0x7f5750078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 -- 172.21.15.26:0/1770468611 shutdown_connections 2024-04-23T18:15:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 -- 172.21.15.26:0/1770468611 wait complete. 2024-04-23T18:15:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.956+0000 7f5755b8b700 1 Processor -- start 2024-04-23T18:15:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 -- start start 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 0x7f57501035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57501026e0 0x7f5750103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750106990 0x7f5750104060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5750111bb0 con 0x7f5750101d20 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5750111a30 con 0x7f5750106990 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f5755b8b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5750111d30 con 0x7f57501026e0 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 0x7f57501035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 0x7f57501035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34460/0 (socket says 172.21.15.26:34460) 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 -- 172.21.15.26:0/4212526897 learned_addr learned my addr 172.21.15.26:0/4212526897 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574ffff700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750106990 0x7f5750104060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574effd700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57501026e0 0x7f5750103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 -- 172.21.15.26:0/4212526897 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57501026e0 msgr2=0x7f5750103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57501026e0 0x7f5750103b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 -- 172.21.15.26:0/4212526897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750106990 msgr2=0x7f5750104060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750106990 0x7f5750104060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.957+0000 7f574f7fe700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57501a7760 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f574f7fe700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 0x7f57501035e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f574000b530 tx=0x7f574000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740015020 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57501a7a50 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57501a7fe0 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5740011ce0 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.958+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740010dd0 con 0x7f5750101d20 2024-04-23T18:15:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.959+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f573c001030 con 0x7f5750101d20 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.960+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5740018020 con 0x7f5750101d20 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.961+0000 7f574cff9700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f57380a3d50 0x7f57380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.961+0000 7f574effd700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f57380a3d50 0x7f57380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.961+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f57400cff10 con 0x7f5750101d20 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.961+0000 7f574cff9700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] conn(0x7f57380ad530 0x7f57380af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.961+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f574001ce70 con 0x7f57380ad530 2024-04-23T18:15:41.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.962+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f57400d03f0 con 0x7f5750101d20 2024-04-23T18:15:41.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.962+0000 7f574ffff700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] conn(0x7f57380ad530 0x7f57380af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:41.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.962+0000 7f574effd700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f57380a3d50 0x7f57380a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f57500790b0 tx=0x7f5744009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:41.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.962+0000 7f574ffff700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] conn(0x7f57380ad530 0x7f57380af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:41.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.963+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== osd.3 v2:172.21.15.26:6810/3619192786 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f574001ce70 con 0x7f57380ad530 2024-04-23T18:15:41.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.980+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 --> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f573c002e60 con 0x7f57380ad530 2024-04-23T18:15:41.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f574cff9700 1 -- 172.21.15.26:0/4212526897 <== osd.3 v2:172.21.15.26:6810/3619192786 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f573c002e60 con 0x7f57380ad530 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] conn(0x7f57380ad530 msgr2=0x7f57380af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] conn(0x7f57380ad530 0x7f57380af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f57380a3d50 msgr2=0x7f57380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f57380a3d50 0x7f57380a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f57500790b0 tx=0x7f5744009190 comp rx=0 tx=0).stop 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 msgr2=0x7f57501035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:41.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 --2- 172.21.15.26:0/4212526897 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5750101d20 0x7f57501035e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f574000b530 tx=0x7f574000b840 comp rx=0 tx=0).stop 2024-04-23T18:15:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f574ffff700 1 -- 172.21.15.26:0/4212526897 reap_dead start 2024-04-23T18:15:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 shutdown_connections 2024-04-23T18:15:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 >> 172.21.15.26:0/4212526897 conn(0x7f57500788f0 msgr2=0x7f575006b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 shutdown_connections 2024-04-23T18:15:41.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:41.981+0000 7f5755b8b700 1 -- 172.21.15.26:0/4212526897 wait complete. 2024-04-23T18:15:42.679 INFO:teuthology.orchestra.run.smithi026.stdout:55834574916 2024-04-23T18:15:42.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T18:15:42.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:42 smithi026 ceph-mon[29364]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:15:43.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:42 smithi110 ceph-mon[32585]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:15:43.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:42 smithi131 ceph-mon[33041]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:15:43.921 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:43 smithi026 ceph-mon[29364]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:43.953 INFO:teuthology.orchestra.run.smithi026.stdout:257698037795 2024-04-23T18:15:43.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T18:15:44.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:43 smithi110 ceph-mon[32585]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:44.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:43 smithi131 ceph-mon[33041]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:45.467 INFO:teuthology.orchestra.run.smithi026.stdout:107374182460 2024-04-23T18:15:45.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T18:15:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 -- 172.21.15.26:0/2615706008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c108a70 msgr2=0x7f1b3c108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 --2- 172.21.15.26:0/2615706008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c108a70 0x7f1b3c108e50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b0a0 tx=0x7f1b2c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 -- 172.21.15.26:0/2615706008 shutdown_connections 2024-04-23T18:15:45.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 --2- 172.21.15.26:0/2615706008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b3c103480 0x7f1b3c10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 --2- 172.21.15.26:0/2615706008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c108a70 0x7f1b3c108e50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 --2- 172.21.15.26:0/2615706008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c102ac0 0x7f1b3c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 -- 172.21.15.26:0/2615706008 >> 172.21.15.26:0/2615706008 conn(0x7f1b3c079bb0 msgr2=0x7f1b3c079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 -- 172.21.15.26:0/2615706008 shutdown_connections 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.227+0000 7f1b4094b700 1 -- 172.21.15.26:0/2615706008 wait complete. 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 Processor -- start 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 -- start start 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b3c102ac0 0x7f1b3c199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 0x7f1b3c199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c108a70 0x7f1b3c19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c112c30 con 0x7f1b3c103480 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c112ab0 con 0x7f1b3c108a70 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b4094b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c112db0 con 0x7f1b3c102ac0 2024-04-23T18:15:45.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.228+0000 7f1b39d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 0x7f1b3c199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 0x7f1b3c199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34472/0 (socket says 172.21.15.26:34472) 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 -- 172.21.15.26:0/1306194749 learned_addr learned my addr 172.21.15.26:0/1306194749 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b3a59c700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b3c102ac0 0x7f1b3c199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b3ad9d700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c108a70 0x7f1b3c19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 -- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b3c102ac0 msgr2=0x7f1b3c199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b3c102ac0 0x7f1b3c199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 -- 172.21.15.26:0/1306194749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c108a70 msgr2=0x7f1b3c19da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c108a70 0x7f1b3c19da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c19e150 con 0x7f1b3c103480 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b39d9b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 0x7f1b3c199680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c920 tx=0x7f1b2400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24007b00 con 0x7f1b3c103480 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b24004ce0 con 0x7f1b3c103480 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3c19e440 con 0x7f1b3c103480 2024-04-23T18:15:45.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24005550 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.229+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3c1a5cc0 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.230+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b3c063110 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.231+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1b24007ca0 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b337fe700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b280a3cd0 0x7f1b280a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b3a59c700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b280a3cd0 0x7f1b280a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1b240d0490 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b337fe700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] conn(0x7f1b280ad4b0 0x7f1b280af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b280aff80 con 0x7f1b280ad4b0 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f1b240d0970 con 0x7f1b3c103480 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b3a59c700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b280a3cd0 0x7f1b280a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009e60 tx=0x7f1b2c00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.232+0000 7f1b3ad9d700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] conn(0x7f1b280ad4b0 0x7f1b280af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.233+0000 7f1b3ad9d700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] conn(0x7f1b280ad4b0 0x7f1b280af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:45.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.234+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== osd.5 v2:172.21.15.131:6808/3617441871 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b280aff80 con 0x7f1b280ad4b0 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.250+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 --> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b3c063590 con 0x7f1b280ad4b0 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b337fe700 1 -- 172.21.15.26:0/1306194749 <== osd.5 v2:172.21.15.131:6808/3617441871 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1b3c063590 con 0x7f1b280ad4b0 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] conn(0x7f1b280ad4b0 msgr2=0x7f1b280af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] conn(0x7f1b280ad4b0 0x7f1b280af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b280a3cd0 msgr2=0x7f1b280a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b280a3cd0 0x7f1b280a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009e60 tx=0x7f1b2c00b7b0 comp rx=0 tx=0).stop 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 msgr2=0x7f1b3c199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:45.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 --2- 172.21.15.26:0/1306194749 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b3c103480 0x7f1b3c199680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c920 tx=0x7f1b2400cc30 comp rx=0 tx=0).stop 2024-04-23T18:15:45.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b3ad9d700 1 -- 172.21.15.26:0/1306194749 reap_dead start 2024-04-23T18:15:45.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 shutdown_connections 2024-04-23T18:15:45.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 >> 172.21.15.26:0/1306194749 conn(0x7f1b3c079bb0 msgr2=0x7f1b3c1018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:45.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 shutdown_connections 2024-04-23T18:15:45.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:44.251+0000 7f1b4094b700 1 -- 172.21.15.26:0/1306194749 wait complete. 2024-04-23T18:15:46.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:45 smithi026 ceph-mon[29364]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:46.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:45 smithi110 ceph-mon[32585]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:46.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:45 smithi131 ceph-mon[33041]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:46.644 INFO:teuthology.orchestra.run.smithi026.stdout:77309411393 2024-04-23T18:15:46.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T18:15:46.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 -- 172.21.15.26:0/93701528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 msgr2=0x7fb95010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 --2- 172.21.15.26:0/93701528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95010f8e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb94800b0a0 tx=0x7fb94800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 -- 172.21.15.26:0/93701528 shutdown_connections 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 --2- 172.21.15.26:0/93701528 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95010f8e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 --2- 172.21.15.26:0/93701528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950108a80 0x7fb950108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 --2- 172.21.15.26:0/93701528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950102af0 0x7fb950102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.856+0000 7fb9558c5700 1 -- 172.21.15.26:0/93701528 >> 172.21.15.26:0/93701528 conn(0x7fb950079ad0 msgr2=0x7fb950079ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.857+0000 7fb9558c5700 1 -- 172.21.15.26:0/93701528 shutdown_connections 2024-04-23T18:15:46.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.857+0000 7fb9558c5700 1 -- 172.21.15.26:0/93701528 wait complete. 2024-04-23T18:15:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.857+0000 7fb9558c5700 1 Processor -- start 2024-04-23T18:15:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.857+0000 7fb9558c5700 1 -- start start 2024-04-23T18:15:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950102af0 0x7fb95019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950108a80 0x7fb95019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb950112c50 con 0x7fb9501034b0 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb950112ad0 con 0x7fb950102af0 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9558c5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb950112dd0 con 0x7fb950108a80 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95019b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34494/0 (socket says 172.21.15.26:34494) 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 -- 172.21.15.26:0/3743915014 learned_addr learned my addr 172.21.15.26:0/3743915014 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94f7fe700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950108a80 0x7fb95019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 -- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950108a80 msgr2=0x7fb95019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950108a80 0x7fb95019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 -- 172.21.15.26:0/3743915014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950102af0 msgr2=0x7fb95019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94effd700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950102af0 0x7fb95019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950102af0 0x7fb95019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9501a02c0 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94f7fe700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb950108a80 0x7fb95019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94effd700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb950102af0 0x7fb95019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb94e7fc700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95019b840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c970 tx=0x7fb94400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.858+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944007aa0 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.859+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb944004ce0 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.859+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9440055c0 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.859+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9501a05b0 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.859+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9501a7e30 con 0x7fb9501034b0 2024-04-23T18:15:46.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.860+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb950101de0 con 0x7fb9501034b0 2024-04-23T18:15:46.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.861+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb944007c40 con 0x7fb9501034b0 2024-04-23T18:15:46.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.862+0000 7fb9548c3700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9380a3d50 0x7fb9380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.862+0000 7fb94effd700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9380a3d50 0x7fb9380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.862+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb9440d0340 con 0x7fb9501034b0 2024-04-23T18:15:46.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.862+0000 7fb9548c3700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] conn(0x7fb9380ad530 0x7fb9380af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.862+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb944010ec0 con 0x7fb9380ad530 2024-04-23T18:15:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.863+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb9440d7050 con 0x7fb9501034b0 2024-04-23T18:15:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.863+0000 7fb94effd700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9380a3d50 0x7fb9380a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb94000baa0 tx=0x7fb940005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.863+0000 7fb94f7fe700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] conn(0x7fb9380ad530 0x7fb9380af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.863+0000 7fb94f7fe700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] conn(0x7fb9380ad530 0x7fb9380af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:46.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.864+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== osd.11 v2:172.21.15.131:6824/2935357051 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb944010ec0 con 0x7fb9380ad530 2024-04-23T18:15:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.882+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 --> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb9501041b0 con 0x7fb9380ad530 2024-04-23T18:15:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9548c3700 1 -- 172.21.15.26:0/3743915014 <== osd.11 v2:172.21.15.131:6824/2935357051 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb9501041b0 con 0x7fb9380ad530 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] conn(0x7fb9380ad530 msgr2=0x7fb9380af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] conn(0x7fb9380ad530 0x7fb9380af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9380a3d50 msgr2=0x7fb9380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb9380a3d50 0x7fb9380a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb94000baa0 tx=0x7fb940005db0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 msgr2=0x7fb95019b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 --2- 172.21.15.26:0/3743915014 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb9501034b0 0x7fb95019b840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c970 tx=0x7fb94400cd30 comp rx=0 tx=0).stop 2024-04-23T18:15:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb94f7fe700 1 -- 172.21.15.26:0/3743915014 reap_dead start 2024-04-23T18:15:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 shutdown_connections 2024-04-23T18:15:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 >> 172.21.15.26:0/3743915014 conn(0x7fb950079ad0 msgr2=0x7fb950101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 shutdown_connections 2024-04-23T18:15:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:46.883+0000 7fb9558c5700 1 -- 172.21.15.26:0/3743915014 wait complete. 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2148815679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 msgr2=0x7fba98107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2148815679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba98107d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b0a0 tx=0x7fba9000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2148815679 shutdown_connections 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2148815679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba98102300 0x7fba9810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2148815679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba98107d70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2148815679 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba98101940 0x7fba98101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2148815679 >> 172.21.15.26:0/2148815679 conn(0x7fba980788f0 msgr2=0x7fba98078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2148815679 shutdown_connections 2024-04-23T18:15:47.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2148815679 wait complete. 2024-04-23T18:15:47.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.072+0000 7fba9d7a0700 1 Processor -- start 2024-04-23T18:15:47.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 -- start start 2024-04-23T18:15:47.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba98101940 0x7fba9819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:47.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba98102300 0x7fba9819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:47.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba981a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fba98111bd0 con 0x7fba98107990 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba98111a50 con 0x7fba98101940 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba9d7a0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fba98111d50 con 0x7fba98102300 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba981a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba981a51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34524/0 (socket says 172.21.15.26:34524) 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 -- 172.21.15.26:0/2484371341 learned_addr learned my addr 172.21.15.26:0/2484371341 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba967fc700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba98102300 0x7fba9819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba96ffd700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba98101940 0x7fba9819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 -- 172.21.15.26:0/2484371341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba98102300 msgr2=0x7fba9819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba98102300 0x7fba9819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 -- 172.21.15.26:0/2484371341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba98101940 msgr2=0x7fba9819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba98101940 0x7fba9819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.073+0000 7fba977fe700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba981a5700 con 0x7fba98107990 2024-04-23T18:15:47.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba977fe700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba981a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fba8800b550 tx=0x7fba8800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88015020 con 0x7fba98107990 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba981a5930 con 0x7fba98107990 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba88011ce0 con 0x7fba98107990 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba981a5e40 con 0x7fba98107990 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.074+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88010c90 con 0x7fba98107990 2024-04-23T18:15:47.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.075+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba980630d0 con 0x7fba98107990 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.076+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fba88018020 con 0x7fba98107990 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba8ffff700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fba840a3cd0 0x7fba840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba96ffd700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fba840a3cd0 0x7fba840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fba880d0140 con 0x7fba98107990 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba8ffff700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] conn(0x7fba840ad4b0 0x7fba840af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba88018ec0 con 0x7fba840ad4b0 2024-04-23T18:15:47.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fba880d04f0 con 0x7fba98107990 2024-04-23T18:15:47.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba967fc700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] conn(0x7fba840ad4b0 0x7fba840af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:47.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba96ffd700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fba840a3cd0 0x7fba840a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fd0 tx=0x7fba9000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:47.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.077+0000 7fba967fc700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] conn(0x7fba840ad4b0 0x7fba840af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:47.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.078+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== osd.6 v2:172.21.15.26:6818/3473982661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba88018ec0 con 0x7fba840ad4b0 2024-04-23T18:15:47.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.094+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 --> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba980634d0 con 0x7fba840ad4b0 2024-04-23T18:15:47.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba8ffff700 1 -- 172.21.15.26:0/2484371341 <== osd.6 v2:172.21.15.26:6818/3473982661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba980634d0 con 0x7fba840ad4b0 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] conn(0x7fba840ad4b0 msgr2=0x7fba840af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] conn(0x7fba840ad4b0 0x7fba840af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fba840a3cd0 msgr2=0x7fba840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fba840a3cd0 0x7fba840a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fd0 tx=0x7fba9000b7b0 comp rx=0 tx=0).stop 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 msgr2=0x7fba981a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 --2- 172.21.15.26:0/2484371341 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fba98107990 0x7fba981a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fba8800b550 tx=0x7fba8800b860 comp rx=0 tx=0).stop 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba977fe700 1 -- 172.21.15.26:0/2484371341 reap_dead start 2024-04-23T18:15:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 shutdown_connections 2024-04-23T18:15:47.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 >> 172.21.15.26:0/2484371341 conn(0x7fba980788f0 msgr2=0x7fba981007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:47.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 shutdown_connections 2024-04-23T18:15:47.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:47.095+0000 7fba9d7a0700 1 -- 172.21.15.26:0/2484371341 wait complete. 2024-04-23T18:15:47.619 INFO:teuthology.orchestra.run.smithi026.stdout:167503724595 2024-04-23T18:15:47.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T18:15:48.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:47 smithi110 ceph-mon[32585]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:48.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:47 smithi026 ceph-mon[29364]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:48.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:47 smithi131 ceph-mon[33041]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:48.713 INFO:teuthology.orchestra.run.smithi026.stdout:347892350998 2024-04-23T18:15:48.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T18:15:49.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 -- 172.21.15.26:0/956814168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc102a90 msgr2=0x7ff7bc102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 --2- 172.21.15.26:0/956814168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc102a90 0x7ff7bc102f10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009990 tx=0x7ff7b0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 -- 172.21.15.26:0/956814168 shutdown_connections 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 --2- 172.21.15.26:0/956814168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff7bc103450 0x7ff7bc10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 --2- 172.21.15.26:0/956814168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7bc108a50 0x7ff7bc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 --2- 172.21.15.26:0/956814168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc102a90 0x7ff7bc102f10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 -- 172.21.15.26:0/956814168 >> 172.21.15.26:0/956814168 conn(0x7ff7bc079ba0 msgr2=0x7ff7bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 -- 172.21.15.26:0/956814168 shutdown_connections 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.745+0000 7ff7c1572700 1 -- 172.21.15.26:0/956814168 wait complete. 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 Processor -- start 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 -- start start 2024-04-23T18:15:49.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7bc102a90 0x7ff7bc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 0x7ff7bc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff7bc108a50 0x7ff7bc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc112c80 con 0x7ff7bc103450 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc112b00 con 0x7ff7bc102a90 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.746+0000 7ff7c1572700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc112e00 con 0x7ff7bc108a50 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 0x7ff7bc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 0x7ff7bc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33074/0 (socket says 172.21.15.26:33074) 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 -- 172.21.15.26:0/340978997 learned_addr learned my addr 172.21.15.26:0/340978997 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7baffd700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7bc102a90 0x7ff7bc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7bb7fe700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff7bc108a50 0x7ff7bc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 -- 172.21.15.26:0/340978997 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff7bc108a50 msgr2=0x7ff7bc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff7bc108a50 0x7ff7bc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 -- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7bc102a90 msgr2=0x7ff7bc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7bc102a90 0x7ff7bc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc1a7700 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7ba7fc700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 0x7ff7bc075280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0000c00 tx=0x7ff7b00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0027070 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc1a7990 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7b00040e0 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc1a7ec0 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.747+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b000f460 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.748+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7bc0630d0 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.749+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff7b002a070 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7a3fff700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff7a40a3cd0 0x7ff7a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7baffd700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff7a40a3cd0 0x7ff7a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff7b00d2180 con 0x7ff7bc103450 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7a3fff700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] conn(0x7ff7a40ad4b0 0x7ff7a40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7a40aff80 con 0x7ff7a40ad4b0 2024-04-23T18:15:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.750+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff7b00d2530 con 0x7ff7bc103450 2024-04-23T18:15:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.751+0000 7ff7baffd700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff7a40a3cd0 0x7ff7a40a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009a10 tx=0x7ff7ac009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.751+0000 7ff7bb7fe700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] conn(0x7ff7a40ad4b0 0x7ff7a40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.751+0000 7ff7bb7fe700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] conn(0x7ff7a40ad4b0 0x7ff7a40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.752+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== osd.4 v2:172.21.15.110:6808/2989079623 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a40aff80 con 0x7ff7a40ad4b0 2024-04-23T18:15:49.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 --> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7bc0634d0 con 0x7ff7a40ad4b0 2024-04-23T18:15:49.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7a3fff700 1 -- 172.21.15.26:0/340978997 <== osd.4 v2:172.21.15.110:6808/2989079623 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff7bc0634d0 con 0x7ff7a40ad4b0 2024-04-23T18:15:49.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] conn(0x7ff7a40ad4b0 msgr2=0x7ff7a40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] conn(0x7ff7a40ad4b0 0x7ff7a40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff7a40a3cd0 msgr2=0x7ff7a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff7a40a3cd0 0x7ff7a40a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009a10 tx=0x7ff7ac009190 comp rx=0 tx=0).stop 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 msgr2=0x7ff7bc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 --2- 172.21.15.26:0/340978997 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff7bc103450 0x7ff7bc075280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0000c00 tx=0x7ff7b00047b0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7bb7fe700 1 -- 172.21.15.26:0/340978997 reap_dead start 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 shutdown_connections 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.769+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 >> 172.21.15.26:0/340978997 conn(0x7ff7bc079ba0 msgr2=0x7ff7bc101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.770+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 shutdown_connections 2024-04-23T18:15:49.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:49.770+0000 7ff7c1572700 1 -- 172.21.15.26:0/340978997 wait complete. 2024-04-23T18:15:50.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:49 smithi110 ceph-mon[32585]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:49 smithi026 ceph-mon[29364]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:50.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:49 smithi131 ceph-mon[33041]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:15:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 -- 172.21.15.26:0/2852780473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 msgr2=0x7f9730101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 --2- 172.21.15.26:0/2852780473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730101de0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9724009990 tx=0x7f9724009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 -- 172.21.15.26:0/2852780473 shutdown_connections 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 --2- 172.21.15.26:0/2852780473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 0x7f973010e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 --2- 172.21.15.26:0/2852780473 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 0x7f9730107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 --2- 172.21.15.26:0/2852780473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730101de0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 -- 172.21.15.26:0/2852780473 >> 172.21.15.26:0/2852780473 conn(0x7f97300789e0 msgr2=0x7f9730078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 -- 172.21.15.26:0/2852780473 shutdown_connections 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.450+0000 7f97366a9700 1 -- 172.21.15.26:0/2852780473 wait complete. 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.451+0000 7f97366a9700 1 Processor -- start 2024-04-23T18:15:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.451+0000 7f97366a9700 1 -- start start 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.451+0000 7f97366a9700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 0x7f973019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 0x7f97301a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9730111ba0 con 0x7f9730101960 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9730111a20 con 0x7f9730102320 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9730111d20 con 0x7f9730107960 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33098/0 (socket says 172.21.15.26:33098) 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 -- 172.21.15.26:0/779921251 learned_addr learned my addr 172.21.15.26:0/779921251 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f9734c46700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 0x7f97301a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972f7fe700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 0x7f973019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 -- 172.21.15.26:0/779921251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 msgr2=0x7f97301a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 0x7f97301a4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 -- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 msgr2=0x7f973019a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 0x7f973019a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97301a54a0 con 0x7f9730101960 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972f7fe700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9730102320 0x7f973019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f9734c46700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9730107960 0x7f97301a4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972ffff700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730199ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f972000c970 tx=0x7f972000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720007aa0 con 0x7f9730101960 2024-04-23T18:15:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97301a5730 con 0x7f9730101960 2024-04-23T18:15:50.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9720004ce0 con 0x7f9730101960 2024-04-23T18:15:50.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.452+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97200055c0 con 0x7f9730101960 2024-04-23T18:15:50.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.453+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97301a5bf0 con 0x7f9730101960 2024-04-23T18:15:50.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.453+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9730063150 con 0x7f9730101960 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9720007c40 con 0x7f9730101960 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97180a3cd0 0x7f97180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972f7fe700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97180a3cd0 0x7f97180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f97200d0620 con 0x7f9730101960 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] conn(0x7f97180ad4b0 0x7f97180af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9720021e50 con 0x7f97180ad4b0 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.455+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f97200d0b00 con 0x7f9730101960 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.456+0000 7f972f7fe700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97180a3cd0 0x7f97180a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9724006010 tx=0x7f972400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:50.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.456+0000 7f9734c46700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] conn(0x7f97180ad4b0 0x7f97180af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:50.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.456+0000 7f9734c46700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] conn(0x7f97180ad4b0 0x7f97180af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:50.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.457+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== osd.10 v2:172.21.15.110:6824/3721710619 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9720021e50 con 0x7f97180ad4b0 2024-04-23T18:15:50.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.473+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 --> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9730063580 con 0x7f97180ad4b0 2024-04-23T18:15:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f972d7fa700 1 -- 172.21.15.26:0/779921251 <== osd.10 v2:172.21.15.110:6824/3721710619 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9730063580 con 0x7f97180ad4b0 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] conn(0x7f97180ad4b0 msgr2=0x7f97180af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] conn(0x7f97180ad4b0 0x7f97180af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97180a3cd0 msgr2=0x7f97180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97180a3cd0 0x7f97180a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9724006010 tx=0x7f972400b540 comp rx=0 tx=0).stop 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 msgr2=0x7f9730199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f97366a9700 1 --2- 172.21.15.26:0/779921251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9730101960 0x7f9730199ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f972000c970 tx=0x7f972000cd30 comp rx=0 tx=0).stop 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.474+0000 7f9734c46700 1 -- 172.21.15.26:0/779921251 reap_dead start 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.475+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 shutdown_connections 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.475+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 >> 172.21.15.26:0/779921251 conn(0x7f97300789e0 msgr2=0x7f9730104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.475+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 shutdown_connections 2024-04-23T18:15:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:50.475+0000 7f97366a9700 1 -- 172.21.15.26:0/779921251 wait complete. 2024-04-23T18:15:51.834 INFO:teuthology.orchestra.run.smithi026.stdout:141733920825 2024-04-23T18:15:51.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T18:15:52.138 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:51 smithi026 ceph-mon[29364]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:52.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:51 smithi110 ceph-mon[32585]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:52.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:51 smithi131 ceph-mon[33041]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:15:52.469 INFO:teuthology.orchestra.run.smithi026.stdout:313532612634 2024-04-23T18:15:52.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.681+0000 7f19dea1c700 1 -- 172.21.15.26:0/1465693423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8108ab0 msgr2=0x7f19d8108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.681+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1465693423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8108ab0 0x7f19d8108e90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009990 tx=0x7f19c8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 -- 172.21.15.26:0/1465693423 shutdown_connections 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1465693423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19d8103480 0x7f19d810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1465693423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8108ab0 0x7f19d8108e90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1465693423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19d8102ac0 0x7f19d8102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 -- 172.21.15.26:0/1465693423 >> 172.21.15.26:0/1465693423 conn(0x7f19d8079bf0 msgr2=0x7f19d807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 -- 172.21.15.26:0/1465693423 shutdown_connections 2024-04-23T18:15:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 -- 172.21.15.26:0/1465693423 wait complete. 2024-04-23T18:15:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 Processor -- start 2024-04-23T18:15:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.682+0000 7f19dea1c700 1 -- start start 2024-04-23T18:15:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19d8102ac0 0x7f19d8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 0x7f19d8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19d8108ab0 0x7f19d80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8112cb0 con 0x7f19d8103480 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8112b30 con 0x7f19d8102ac0 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dea1c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8112e30 con 0x7f19d8108ab0 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 0x7f19d8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19d77fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 0x7f19d8075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33126/0 (socket says 172.21.15.26:33126) 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19d77fe700 1 -- 172.21.15.26:0/1318223136 learned_addr learned my addr 172.21.15.26:0/1318223136 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19d7fff700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19d8102ac0 0x7f19d8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.683+0000 7f19dcfb9700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19d8108ab0 0x7f19d80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 -- 172.21.15.26:0/1318223136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19d8108ab0 msgr2=0x7f19d80757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19d8108ab0 0x7f19d80757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 -- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19d8102ac0 msgr2=0x7f19d8074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19d8102ac0 0x7f19d8074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d81a7670 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d77fe700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 0x7f19d8075280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00c9b0 tx=0x7f19cc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc007b00 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19cc00ce50 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d81a7960 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc018660 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.684+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d81a7ef0 con 0x7f19d8103480 2024-04-23T18:15:52.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.685+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19d8063110 con 0x7f19d8103480 2024-04-23T18:15:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.687+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f19cc00f450 con 0x7f19d8103480 2024-04-23T18:15:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.687+0000 7f19d57fa700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f19c00a3d50 0x7f19c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.687+0000 7f19d7fff700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f19c00a3d50 0x7f19c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f19cc0d08e0 con 0x7f19d8103480 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19d57fa700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] conn(0x7f19c00ad530 0x7f19c00af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19c00b0000 con 0x7f19c00ad530 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f19cc0d0dc0 con 0x7f19d8103480 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19d7fff700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f19c00a3d50 0x7f19c00a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b5c0 tx=0x7f19c8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.688+0000 7f19dcfb9700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] conn(0x7f19c00ad530 0x7f19c00af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.689+0000 7f19dcfb9700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] conn(0x7f19c00ad530 0x7f19c00af950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.690+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== osd.7 v2:172.21.15.110:6816/3458351312 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f19c00b0000 con 0x7f19c00ad530 2024-04-23T18:15:52.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.706+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 --> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f19d8063540 con 0x7f19c00ad530 2024-04-23T18:15:52.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.706+0000 7f19d57fa700 1 -- 172.21.15.26:0/1318223136 <== osd.7 v2:172.21.15.110:6816/3458351312 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f19d8063540 con 0x7f19c00ad530 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] conn(0x7f19c00ad530 msgr2=0x7f19c00af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] conn(0x7f19c00ad530 0x7f19c00af950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f19c00a3d50 msgr2=0x7f19c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f19c00a3d50 0x7f19c00a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b5c0 tx=0x7f19c8005960 comp rx=0 tx=0).stop 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 msgr2=0x7f19d8075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 --2- 172.21.15.26:0/1318223136 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f19d8103480 0x7f19d8075280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00c9b0 tx=0x7f19cc00ccc0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dcfb9700 1 -- 172.21.15.26:0/1318223136 reap_dead start 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 shutdown_connections 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 >> 172.21.15.26:0/1318223136 conn(0x7f19d8079bf0 msgr2=0x7f19d8100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 shutdown_connections 2024-04-23T18:15:52.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:52.707+0000 7f19dea1c700 1 -- 172.21.15.26:0/1318223136 wait complete. 2024-04-23T18:15:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 -- 172.21.15.26:0/158062514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 msgr2=0x7f688406b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 --2- 172.21.15.26:0/158062514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f688406b580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6878009990 tx=0x7f6878009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 -- 172.21.15.26:0/158062514 shutdown_connections 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 --2- 172.21.15.26:0/158062514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406bac0 0x7f688410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 --2- 172.21.15.26:0/158062514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6884108a30 0x7f6884108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 --2- 172.21.15.26:0/158062514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f688406b580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 -- 172.21.15.26:0/158062514 >> 172.21.15.26:0/158062514 conn(0x7f68840788b0 msgr2=0x7f6884078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 -- 172.21.15.26:0/158062514 shutdown_connections 2024-04-23T18:15:53.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.294+0000 7f688c54c700 1 -- 172.21.15.26:0/158062514 wait complete. 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 Processor -- start 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 -- start start 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f6884199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f688406bac0 0x7f6884199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6884108a30 0x7f688419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6884112bd0 con 0x7f688406b120 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6884112a50 con 0x7f6884108a30 2024-04-23T18:15:53.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.295+0000 7f688c54c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6884112d50 con 0x7f688406bac0 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f6884199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f6884199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33146/0 (socket says 172.21.15.26:33146) 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 -- 172.21.15.26:0/1135352477 learned_addr learned my addr 172.21.15.26:0/1135352477 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f6889ae7700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f688406bac0 0x7f6884199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688aae9700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6884108a30 0x7f688419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 -- 172.21.15.26:0/1135352477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f688406bac0 msgr2=0x7f6884199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f688406bac0 0x7f6884199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 -- 172.21.15.26:0/1135352477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6884108a30 msgr2=0x7f688419d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6884108a30 0x7f688419d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688419e110 con 0x7f688406b120 2024-04-23T18:15:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688a2e8700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f6884199100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f68740060b0 tx=0x7f687400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687401a070 con 0x7f688406b120 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f688419e400 con 0x7f688406b120 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f687400ddd0 con 0x7f688406b120 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874014600 con 0x7f688406b120 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.296+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68841a5c80 con 0x7f688406b120 2024-04-23T18:15:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.297+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68840631d0 con 0x7f688406b120 2024-04-23T18:15:53.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.299+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f687401d070 con 0x7f688406b120 2024-04-23T18:15:53.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.299+0000 7f68837fe700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f686c0a3cd0 0x7f686c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:53.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.299+0000 7f6889ae7700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f686c0a3cd0 0x7f686c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:53.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.299+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f68740d02d0 con 0x7f688406b120 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.299+0000 7f68837fe700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] conn(0x7f686c0ad4b0 0x7f686c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.300+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6874013ec0 con 0x7f686c0ad4b0 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.300+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f68740d07b0 con 0x7f688406b120 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.300+0000 7f688aae9700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] conn(0x7f686c0ad4b0 0x7f686c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.300+0000 7f6889ae7700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f686c0a3cd0 0x7f686c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68780098e0 tx=0x7f6878005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.300+0000 7f688aae9700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] conn(0x7f686c0ad4b0 0x7f686c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:53.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.301+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== osd.9 v2:172.21.15.26:6826/1397303359 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6874013ec0 con 0x7f686c0ad4b0 2024-04-23T18:15:53.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.317+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 --> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f68840635d0 con 0x7f686c0ad4b0 2024-04-23T18:15:53.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f68837fe700 1 -- 172.21.15.26:0/1135352477 <== osd.9 v2:172.21.15.26:6826/1397303359 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f68840635d0 con 0x7f686c0ad4b0 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] conn(0x7f686c0ad4b0 msgr2=0x7f686c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] conn(0x7f686c0ad4b0 0x7f686c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f686c0a3cd0 msgr2=0x7f686c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f686c0a3cd0 0x7f686c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68780098e0 tx=0x7f6878005960 comp rx=0 tx=0).stop 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 msgr2=0x7f6884199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:53.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 --2- 172.21.15.26:0/1135352477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f688406b120 0x7f6884199100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f68740060b0 tx=0x7f687400d3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688aae9700 1 -- 172.21.15.26:0/1135352477 reap_dead start 2024-04-23T18:15:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 shutdown_connections 2024-04-23T18:15:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 >> 172.21.15.26:0/1135352477 conn(0x7f68840788b0 msgr2=0x7f68840ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 shutdown_connections 2024-04-23T18:15:53.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:53.318+0000 7f688c54c700 1 -- 172.21.15.26:0/1135352477 wait complete. 2024-04-23T18:15:53.561 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:54.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:53 smithi026 ceph-mon[29364]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:15:54.160 INFO:teuthology.orchestra.run.smithi026.stdout:201863462958 2024-04-23T18:15:54.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T18:15:54.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:53 smithi110 ceph-mon[32585]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:15:54.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:53 smithi131 ceph-mon[33041]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:15:54.723 INFO:teuthology.orchestra.run.smithi026.stdout:223338299434 2024-04-23T18:15:54.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T18:15:56.001 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:55 smithi026 ceph-mon[29364]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:15:56.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:55 smithi110 ceph-mon[32585]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:15:56.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:55 smithi131 ceph-mon[33041]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:15:56.533 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.533 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.534 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:56.585 INFO:teuthology.orchestra.run.smithi026.stdout:292057776161 2024-04-23T18:15:56.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T18:15:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 -- 172.21.15.26:0/1609815928 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c103450 msgr2=0x7f8b5c10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1609815928 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c103450 0x7f8b5c10f8b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400b0a0 tx=0x7f8b5400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 -- 172.21.15.26:0/1609815928 shutdown_connections 2024-04-23T18:15:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1609815928 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c103450 0x7f8b5c10f8b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1609815928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c108a50 0x7f8b5c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1609815928 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c102a90 0x7f8b5c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 -- 172.21.15.26:0/1609815928 >> 172.21.15.26:0/1609815928 conn(0x7f8b5c079ba0 msgr2=0x7f8b5c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 -- 172.21.15.26:0/1609815928 shutdown_connections 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.991+0000 7f8b6104a700 1 -- 172.21.15.26:0/1609815928 wait complete. 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 Processor -- start 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 -- start start 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 0x7f8b5c199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c103450 0x7f8b5c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c112be0 con 0x7f8b5c108a50 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c112a60 con 0x7f8b5c102a90 2024-04-23T18:15:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b6104a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c112d60 con 0x7f8b5c103450 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b5b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b5b59e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49078/0 (socket says 172.21.15.26:49078) 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b5b59e700 1 -- 172.21.15.26:0/1325118915 learned_addr learned my addr 172.21.15.26:0/1325118915 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b5a59c700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c103450 0x7f8b5c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.992+0000 7f8b5ad9d700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 0x7f8b5c199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 -- 172.21.15.26:0/1325118915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c103450 msgr2=0x7f8b5c199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c103450 0x7f8b5c199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 -- 172.21.15.26:0/1325118915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 msgr2=0x7f8b5c199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 0x7f8b5c199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 -- 172.21.15.26:0/1325118915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5c19e040 con 0x7f8b5c108a50 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5ad9d700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 0x7f8b5c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b5b59e700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400b830 tx=0x7f8b54004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:57.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54021070 con 0x7f8b5c108a50 2024-04-23T18:15:57.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5c19e2d0 con 0x7f8b5c108a50 2024-04-23T18:15:57.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b54007e30 con 0x7f8b5c108a50 2024-04-23T18:15:57.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540047d0 con 0x7f8b5c108a50 2024-04-23T18:15:57.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.993+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b5c1a5bb0 con 0x7f8b5c108a50 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.994+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b5c069730 con 0x7f8b5c108a50 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.995+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8b54007980 con 0x7f8b5c108a50 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.996+0000 7f8b43fff700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 0x7f8b440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.996+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8b5400f020 con 0x7f8b5c108a50 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.996+0000 7f8b5ad9d700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 0x7f8b440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:57.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:57.996+0000 7f8b5ad9d700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 0x7f8b440a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c005fd0 tx=0x7f8b4c00a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:57.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.001+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b5402a020 con 0x7f8b5c108a50 2024-04-23T18:15:58.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:57 smithi110 ceph-mon[32585]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:15:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.209+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8b5c19a390 con 0x7f8b5c108a50 2024-04-23T18:15:58.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:57 smithi026 ceph-mon[29364]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:15:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.210+0000 7f8b43fff700 1 -- 172.21.15.26:0/1325118915 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8b5409e400 con 0x7f8b5c108a50 2024-04-23T18:15:58.203 INFO:teuthology.orchestra.run.smithi026.stdout:34359738445 2024-04-23T18:15:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 msgr2=0x7f8b440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 0x7f8b440a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c005fd0 tx=0x7f8b4c00a5e0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 msgr2=0x7f8b5c19d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400b830 tx=0x7f8b54004ce0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 shutdown_connections 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8b440a3cd0 0x7f8b440a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8b5c108a50 0x7f8b5c19d900 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b5c102a90 0x7f8b5c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 --2- 172.21.15.26:0/1325118915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8b5c103450 0x7f8b5c199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 >> 172.21.15.26:0/1325118915 conn(0x7f8b5c079ba0 msgr2=0x7f8b5c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 shutdown_connections 2024-04-23T18:15:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:15:58.212+0000 7f8b6104a700 1 -- 172.21.15.26:0/1325118915 wait complete. 2024-04-23T18:15:58.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:57 smithi131 ceph-mon[33041]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:15:59.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:58 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1325118915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:15:59.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:58 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1325118915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:15:59.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:58 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1325118915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:16:00.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:15:59 smithi110 ceph-mon[32585]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:00.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:15:59 smithi026 ceph-mon[29364]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:00.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:15:59 smithi131 ceph-mon[33041]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:01.260 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:02.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:01 smithi110 ceph-mon[32585]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:01 smithi026 ceph-mon[29364]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:02.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:01 smithi131 ceph-mon[33041]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:03.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738445 for osd.0 2024-04-23T18:16:03.223 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 -- 172.21.15.26:0/831276575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 msgr2=0x7f9b6406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/831276575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b6406b440 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9b58009990 tx=0x7f9b58009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 -- 172.21.15.26:0/831276575 shutdown_connections 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/831276575 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 0x7f9b6410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/831276575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 0x7f9b64108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/831276575 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b6406b440 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 -- 172.21.15.26:0/831276575 >> 172.21.15.26:0/831276575 conn(0x7f9b640788b0 msgr2=0x7f9b64078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 -- 172.21.15.26:0/831276575 shutdown_connections 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.894+0000 7f9b6ac61700 1 -- 172.21.15.26:0/831276575 wait complete. 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 Processor -- start 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 -- start start 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 0x7f9b64101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 0x7f9b64102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9b64112be0 con 0x7f9b6406afe0 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9b64112a60 con 0x7f9b64108a20 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b6ac61700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9b64112d60 con 0x7f9b6406b980 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b689fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b689fd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49094/0 (socket says 172.21.15.26:49094) 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.895+0000 7f9b689fd700 1 -- 172.21.15.26:0/2188807119 learned_addr learned my addr 172.21.15.26:0/2188807119 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b63fff700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 0x7f9b64101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b691fe700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 0x7f9b64102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 -- 172.21.15.26:0/2188807119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 msgr2=0x7f9b64101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 0x7f9b64101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 -- 172.21.15.26:0/2188807119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 msgr2=0x7f9b64102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 0x7f9b64102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 -- 172.21.15.26:0/2188807119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b641a7600 con 0x7f9b6406afe0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b689fd700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0060b0 tx=0x7f9b5c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c01a070 con 0x7f9b6406afe0 2024-04-23T18:16:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b5c00dd20 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b641a78f0 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c014600 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.896+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b641a7e00 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.897+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b64069730 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.898+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9b5c01d070 con 0x7f9b6406afe0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.899+0000 7f9b61ffb700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 0x7f9b4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.899+0000 7f9b63fff700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 0x7f9b4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.899+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9b5c0cfb30 con 0x7f9b6406afe0 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.900+0000 7f9b63fff700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 0x7f9b4c0a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800b2e0 tx=0x7f9b58005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:03.904+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b5c010070 con 0x7f9b6406afe0 2024-04-23T18:16:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.117+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f9b6402df80 con 0x7f9b6406afe0 2024-04-23T18:16:04.112 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:03 smithi026 ceph-mon[29364]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.117+0000 7f9b61ffb700 1 -- 172.21.15.26:0/2188807119 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f9b5c09c560 con 0x7f9b6406afe0 2024-04-23T18:16:04.113 INFO:teuthology.orchestra.run.smithi026.stdout:347892351001 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 msgr2=0x7f9b4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 0x7f9b4c0a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800b2e0 tx=0x7f9b58005960 comp rx=0 tx=0).stop 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 msgr2=0x7f9b64101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0060b0 tx=0x7f9b5c00d3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 shutdown_connections 2024-04-23T18:16:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9b4c0a3cd0 0x7f9b4c0a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b64108a20 0x7f9b64102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9b6406afe0 0x7f9b64101a00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 --2- 172.21.15.26:0/2188807119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b6406b980 0x7f9b64101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.119+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 >> 172.21.15.26:0/2188807119 conn(0x7f9b640788b0 msgr2=0x7f9b640ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.120+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 shutdown_connections 2024-04-23T18:16:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.120+0000 7f9b6ac61700 1 -- 172.21.15.26:0/2188807119 wait complete. 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.120+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2820270572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d18108a30 msgr2=0x7f8d18108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.120+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2820270572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d18108a30 0x7f8d18108e10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009990 tx=0x7f8d08009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2820270572 shutdown_connections 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2820270572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d1806b0f0 0x7f8d1810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2820270572 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d18108a30 0x7f8d18108e10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2820270572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806a750 0x7f8d1806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2820270572 >> 172.21.15.26:0/2820270572 conn(0x7f8d180788f0 msgr2=0x7f8d18078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2820270572 shutdown_connections 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2820270572 wait complete. 2024-04-23T18:16:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 Processor -- start 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.121+0000 7f8d1ce6a700 1 -- start start 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806b0f0 0x7f8d18199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d18108a30 0x7f8d1819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18112c30 con 0x7f8d1806a750 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18112ab0 con 0x7f8d18108a30 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1ce6a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18112db0 con 0x7f8d1806b0f0 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49110/0 (socket says 172.21.15.26:49110) 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 -- 172.21.15.26:0/2301256185 learned_addr learned my addr 172.21.15.26:0/2301256185 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d15d9b700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806b0f0 0x7f8d18199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d16d9d700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d18108a30 0x7f8d1819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 -- 172.21.15.26:0/2301256185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806b0f0 msgr2=0x7f8d18199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806b0f0 0x7f8d18199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 -- 172.21.15.26:0/2301256185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d18108a30 msgr2=0x7f8d1819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d18108a30 0x7f8d1819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 -- 172.21.15.26:0/2301256185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1819e100 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d1659c700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d08000c00 tx=0x7f8d080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.122+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08027070 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.123+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1819e390 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.123+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d080040e0 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.123+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800f460 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.123+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d181a5c70 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.124+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d18069730 con 0x7f8d1806a750 2024-04-23T18:16:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.126+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8d0802a070 con 0x7f8d1806a750 2024-04-23T18:16:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.126+0000 7f8d077fe700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 0x7f8d000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.126+0000 7f8d15d9b700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 0x7f8d000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.126+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8d080d17f0 con 0x7f8d1806a750 2024-04-23T18:16:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.127+0000 7f8d15d9b700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 0x7f8d000a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d18078fd0 tx=0x7f8d0c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.130+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d08031070 con 0x7f8d1806a750 2024-04-23T18:16:04.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:03 smithi110 ceph-mon[32585]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:04.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:03 smithi131 ceph-mon[33041]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.345+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8d1802df80 con 0x7f8d1806a750 2024-04-23T18:16:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.345+0000 7f8d077fe700 1 -- 172.21.15.26:0/2301256185 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d08013070 con 0x7f8d1806a750 2024-04-23T18:16:04.340 INFO:teuthology.orchestra.run.smithi026.stdout:55834574921 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 msgr2=0x7f8d000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 0x7f8d000a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d18078fd0 tx=0x7f8d0c00a380 comp rx=0 tx=0).stop 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 msgr2=0x7f8d18199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d08000c00 tx=0x7f8d080047b0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 shutdown_connections 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f8d000a3cd0 0x7f8d000a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d18108a30 0x7f8d1819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1806a750 0x7f8d18199140 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 --2- 172.21.15.26:0/2301256185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1806b0f0 0x7f8d18199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 >> 172.21.15.26:0/2301256185 conn(0x7f8d180788f0 msgr2=0x7f8d180ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 shutdown_connections 2024-04-23T18:16:04.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:04.348+0000 7f8d1ce6a700 1 -- 172.21.15.26:0/2301256185 wait complete. 2024-04-23T18:16:05.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2188807119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:16:05.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2301256185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:16:05.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:04 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:05.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2188807119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:16:05.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2301256185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:16:05.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:04 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:05.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:04 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2188807119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:16:05.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:04 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2301256185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:16:05.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:04 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:06.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:05 smithi110 ceph-mon[32585]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:06.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:05 smithi026 ceph-mon[29364]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:06.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:05 smithi131 ceph-mon[33041]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:07.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:07 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:07 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:07 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:07 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:07 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1508814480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:16:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:07.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:07 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:07 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:07 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:07 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:07 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1508814480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 -- 172.21.15.26:0/1245778901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c102300 msgr2=0x7f0e7c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 --2- 172.21.15.26:0/1245778901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c102300 0x7f0e7c10e7f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400b0a0 tx=0x7f0e7400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 -- 172.21.15.26:0/1245778901 shutdown_connections 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 --2- 172.21.15.26:0/1245778901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c102300 0x7f0e7c10e7f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 --2- 172.21.15.26:0/1245778901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c107990 0x7f0e7c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 --2- 172.21.15.26:0/1245778901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c101940 0x7f0e7c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 -- 172.21.15.26:0/1245778901 >> 172.21.15.26:0/1245778901 conn(0x7f0e7c0788f0 msgr2=0x7f0e7c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 -- 172.21.15.26:0/1245778901 shutdown_connections 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.538+0000 7f0e81539700 1 -- 172.21.15.26:0/1245778901 wait complete. 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 Processor -- start 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 -- start start 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 0x7f0e7c103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 0x7f0e7c103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c111b20 con 0x7f0e7c101940 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c1119a0 con 0x7f0e7c102300 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e81539700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c111ca0 con 0x7f0e7c107990 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49142/0 (socket says 172.21.15.26:49142) 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 -- 172.21.15.26:0/2878159527 learned_addr learned my addr 172.21.15.26:0/2878159527 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7b7fe700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 0x7f0e7c103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:07.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7a7fc700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 0x7f0e7c103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 -- 172.21.15.26:0/2878159527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 msgr2=0x7f0e7c103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 0x7f0e7c103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 -- 172.21.15.26:0/2878159527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 msgr2=0x7f0e7c103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 0x7f0e7c103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7affd700 1 -- 172.21.15.26:0/2878159527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c1a75c0 con 0x7f0e7c101940 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7b7fe700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 0x7f0e7c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.539+0000 7f0e7a7fc700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 0x7f0e7c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e7affd700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00d950 tx=0x7f0e6c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c016070 con 0x7f0e7c101940 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7c1a78b0 con 0x7f0e7c101940 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e6c010460 con 0x7f0e7c101940 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7c1a7e40 con 0x7f0e7c101940 2024-04-23T18:16:07.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.540+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c00f620 con 0x7f0e7c101940 2024-04-23T18:16:07.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.541+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e7c069730 con 0x7f0e7c101940 2024-04-23T18:16:07.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.542+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0e6c019070 con 0x7f0e7c101940 2024-04-23T18:16:07.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.543+0000 7f0e63fff700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 0x7f0e640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:07.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.543+0000 7f0e7a7fc700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 0x7f0e640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:07.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.543+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0e6c0d0020 con 0x7f0e7c101940 2024-04-23T18:16:07.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.544+0000 7f0e7a7fc700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 0x7f0e640a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005fd0 tx=0x7f0e70005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:07.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.552+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e6c09c9d0 con 0x7f0e7c101940 2024-04-23T18:16:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:07 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:07 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:07 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:07 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:07.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:07 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1508814480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:16:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.761+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0e7c02df80 con 0x7f0e7c101940 2024-04-23T18:16:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.762+0000 7f0e63fff700 1 -- 172.21.15.26:0/2878159527 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0e6c09c2d0 con 0x7f0e7c101940 2024-04-23T18:16:07.757 INFO:teuthology.orchestra.run.smithi026.stdout:141733920828 2024-04-23T18:16:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 msgr2=0x7f0e640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 0x7f0e640a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005fd0 tx=0x7f0e70005e20 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 msgr2=0x7f0e7c1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00d950 tx=0x7f0e6c00dd10 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 shutdown_connections 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0e640a3cd0 0x7f0e640a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e7c107990 0x7f0e7c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0e7c101940 0x7f0e7c1034f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 --2- 172.21.15.26:0/2878159527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e7c102300 0x7f0e7c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 >> 172.21.15.26:0/2878159527 conn(0x7f0e7c0788f0 msgr2=0x7f0e7c06aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 shutdown_connections 2024-04-23T18:16:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.764+0000 7f0e81539700 1 -- 172.21.15.26:0/2878159527 wait complete. 2024-04-23T18:16:08.024 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351001 for osd.11 2024-04-23T18:16:08.024 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:08 smithi026 ceph-mon[29364]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-23T18:16:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:08 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2878159527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:16:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:08 smithi110 ceph-mon[32585]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-23T18:16:08.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:08 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2878159527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:16:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:08 smithi131 ceph-mon[33041]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-23T18:16:08.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:08 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2878159527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:16:08.842 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574916 got 55834574921 for osd.1 2024-04-23T18:16:08.842 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:08.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 -- 172.21.15.26:0/1816760373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498108a70 msgr2=0x7f9498108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 --2- 172.21.15.26:0/1816760373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498108a70 0x7f9498108e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9494009990 tx=0x7f9494009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 -- 172.21.15.26:0/1816760373 shutdown_connections 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 --2- 172.21.15.26:0/1816760373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498103480 0x7f949810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 --2- 172.21.15.26:0/1816760373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498108a70 0x7f9498108e50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 --2- 172.21.15.26:0/1816760373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498102ac0 0x7f9498102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 -- 172.21.15.26:0/1816760373 >> 172.21.15.26:0/1816760373 conn(0x7f9498079bb0 msgr2=0x7f9498079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 -- 172.21.15.26:0/1816760373 shutdown_connections 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 -- 172.21.15.26:0/1816760373 wait complete. 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.991+0000 7f949f285700 1 Processor -- start 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 -- start start 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498103480 0x7f9498199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498108a70 0x7f949819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9498112c40 con 0x7f9498102ac0 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9498112ac0 con 0x7f9498108a70 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949f285700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9498112dc0 con 0x7f9498103480 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49130/0 (socket says 172.21.15.26:49130) 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 -- 172.21.15.26:0/1508814480 learned_addr learned my addr 172.21.15.26:0/1508814480 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949c820700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498103480 0x7f9498199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 -- 172.21.15.26:0/1508814480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498103480 msgr2=0x7f9498199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498103480 0x7f9498199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 -- 172.21.15.26:0/1508814480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498108a70 msgr2=0x7f949819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498108a70 0x7f949819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 -- 172.21.15.26:0/1508814480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949819e140 con 0x7f9498102ac0 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f949d021700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f94940057f0 tx=0x7f9494003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:08.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.992+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494027070 con 0x7f9498102ac0 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.993+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94940040a0 con 0x7f9498102ac0 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.993+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949819e3d0 con 0x7f9498102ac0 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stdout:77309411398 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.993+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494022e20 con 0x7f9498102ac0 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.993+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94981a5cb0 con 0x7f9498102ac0 2024-04-23T18:16:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.993+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9498069730 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.995+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f949402a070 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.995+0000 7f948e7fc700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 0x7f94840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.995+0000 7f949c820700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 0x7f94840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.995+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f94940d1780 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.996+0000 7f949c820700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 0x7f94840a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f949807a290 tx=0x7f948800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:06.999+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9494013070 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.216+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f949802df80 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.216+0000 7f948e7fc700 1 -- 172.21.15.26:0/1508814480 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f949409e240 con 0x7f9498102ac0 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 msgr2=0x7f94840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 0x7f94840a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f949807a290 tx=0x7f948800a300 comp rx=0 tx=0).stop 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 msgr2=0x7f9498199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f94940057f0 tx=0x7f9494003680 comp rx=0 tx=0).stop 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 shutdown_connections 2024-04-23T18:16:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f94840a3cd0 0x7f94840a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498108a70 0x7f949819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9498102ac0 0x7f9498199130 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 --2- 172.21.15.26:0/1508814480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9498103480 0x7f9498199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 >> 172.21.15.26:0/1508814480 conn(0x7f9498079bb0 msgr2=0x7f9498101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 shutdown_connections 2024-04-23T18:16:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:07.218+0000 7f949f285700 1 -- 172.21.15.26:0/1508814480 wait complete. 2024-04-23T18:16:10.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:10 smithi026 ceph-mon[29364]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:16:10.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- 172.21.15.26:0/4083361357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4102af0 msgr2=0x7f49c4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 --2- 172.21.15.26:0/4083361357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4102af0 0x7f49c4102f70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009990 tx=0x7f49b0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- 172.21.15.26:0/4083361357 shutdown_connections 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 --2- 172.21.15.26:0/4083361357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 0x7f49c410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 --2- 172.21.15.26:0/4083361357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4108a80 0x7f49c4108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 --2- 172.21.15.26:0/4083361357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4102af0 0x7f49c4102f70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- 172.21.15.26:0/4083361357 >> 172.21.15.26:0/4083361357 conn(0x7f49c4079b20 msgr2=0x7f49c4079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- 172.21.15.26:0/4083361357 shutdown_connections 2024-04-23T18:16:10.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- 172.21.15.26:0/4083361357 wait complete. 2024-04-23T18:16:10.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 Processor -- start 2024-04-23T18:16:10.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.615+0000 7f49cadd1700 1 -- start start 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4102af0 0x7f49c4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 0x7f49c4075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4112d60 con 0x7f49c4108a80 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4112be0 con 0x7f49c4102af0 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4112ee0 con 0x7f49c41034b0 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59074/0 (socket says 172.21.15.26:59074) 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 -- 172.21.15.26:0/3709711935 learned_addr learned my addr 172.21.15.26:0/3709711935 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c8b6d700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4102af0 0x7f49c4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:10.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c3fff700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 0x7f49c4075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 -- 172.21.15.26:0/3709711935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 msgr2=0x7f49c4075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 0x7f49c4075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 -- 172.21.15.26:0/3709711935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4102af0 msgr2=0x7f49c4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4102af0 0x7f49c4074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 -- 172.21.15.26:0/3709711935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c41a77e0 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c936e700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00b970 tx=0x7f49bc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc019070 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49bc014440 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc00ce60 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.616+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c41a7ad0 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.617+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c41a7fc0 con 0x7f49c4108a80 2024-04-23T18:16:10.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.618+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c4069730 con 0x7f49c4108a80 2024-04-23T18:16:10.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.618+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f49bc01c070 con 0x7f49c4108a80 2024-04-23T18:16:10.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.619+0000 7f49c1ffb700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 0x7f49b40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:10.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.619+0000 7f49c8b6d700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 0x7f49b40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:10.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.619+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f49bc0cf720 con 0x7f49c4108a80 2024-04-23T18:16:10.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.620+0000 7f49c8b6d700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 0x7f49b40a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f49b8006fd0 tx=0x7f49b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:10.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.624+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49bc09c260 con 0x7f49c4108a80 2024-04-23T18:16:10.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:10 smithi110 ceph-mon[32585]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:16:10.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:10 smithi131 ceph-mon[33041]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:16:10.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.841+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f49c402df80 con 0x7f49c4108a80 2024-04-23T18:16:10.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.842+0000 7f49c1ffb700 1 -- 172.21.15.26:0/3709711935 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f49bc09bb60 con 0x7f49c4108a80 2024-04-23T18:16:10.836 INFO:teuthology.orchestra.run.smithi026.stdout:167503724600 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 msgr2=0x7f49b40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 0x7f49b40a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f49b8006fd0 tx=0x7f49b8008040 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 msgr2=0x7f49c40757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00b970 tx=0x7f49bc00bd30 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 shutdown_connections 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f49b40a3cd0 0x7f49b40a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f49c4108a80 0x7f49c40757c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49c4102af0 0x7f49c4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 --2- 172.21.15.26:0/3709711935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49c41034b0 0x7f49c4075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 >> 172.21.15.26:0/3709711935 conn(0x7f49c4079b20 msgr2=0x7f49c4101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 shutdown_connections 2024-04-23T18:16:10.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:10.845+0000 7f49cadd1700 1 -- 172.21.15.26:0/3709711935 wait complete. 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:16:11.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:11 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3709711935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:16:11.643 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:16:11.644 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:11 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3709711935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-23T18:16:11.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:16:11.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:11 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3709711935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:16:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 -- 172.21.15.26:0/2556419265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 msgr2=0x7f1f2406ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 --2- 172.21.15.26:0/2556419265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f2406ab30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009990 tx=0x7f1f20009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 -- 172.21.15.26:0/2556419265 shutdown_connections 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 --2- 172.21.15.26:0/2556419265 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 --2- 172.21.15.26:0/2556419265 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f2406ab30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 --2- 172.21.15.26:0/2556419265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 0x7f1f2410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.241+0000 7f1f2be12700 1 -- 172.21.15.26:0/2556419265 >> 172.21.15.26:0/2556419265 conn(0x7f1f240788f0 msgr2=0x7f1f24078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- 172.21.15.26:0/2556419265 shutdown_connections 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- 172.21.15.26:0/2556419265 wait complete. 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 Processor -- start 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- start start 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 0x7f1f24104160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24114e00 con 0x7f1f2406a750 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24114c80 con 0x7f1f2406b070 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f2be12700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24114f80 con 0x7f1f2410f4f0 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f29bae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.242+0000 7f1f29bae700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59092/0 (socket says 172.21.15.26:59092) 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 -- 172.21.15.26:0/229088780 learned_addr learned my addr 172.21.15.26:0/229088780 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f293ad700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 0x7f1f24104160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f2a3af700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 -- 172.21.15.26:0/229088780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 msgr2=0x7f1f241046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 -- 172.21.15.26:0/229088780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 msgr2=0x7f1f24104160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 0x7f1f24104160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 -- 172.21.15.26:0/229088780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f241ab9e0 con 0x7f1f2406a750 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f2a3af700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f29bae700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000c00 tx=0x7f1f200047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20027070 con 0x7f1f2406a750 2024-04-23T18:16:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f20003f20 con 0x7f1f2406a750 2024-04-23T18:16:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f241abc70 con 0x7f1f2406a750 2024-04-23T18:16:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f241ac0c0 con 0x7f1f2406a750 2024-04-23T18:16:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.243+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20022ce0 con 0x7f1f2406a750 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.244+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f08005330 con 0x7f1f2406a750 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.246+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1f2002a070 con 0x7f1f2406a750 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.246+0000 7f1f1affd700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 0x7f1f100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.246+0000 7f1f293ad700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 0x7f1f100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.246+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1f200d1400 con 0x7f1f2406a750 2024-04-23T18:16:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.247+0000 7f1f293ad700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 0x7f1f100a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f241053e0 tx=0x7f1f1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.250+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f20013070 con 0x7f1f2406a750 2024-04-23T18:16:12.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.468+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f1f080058f0 con 0x7f1f2406a750 2024-04-23T18:16:12.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.468+0000 7f1f1affd700 1 -- 172.21.15.26:0/229088780 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1f2009dd20 con 0x7f1f2406a750 2024-04-23T18:16:12.463 INFO:teuthology.orchestra.run.smithi026.stdout:257698037802 2024-04-23T18:16:12.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 msgr2=0x7f1f100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:12.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 0x7f1f100a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f241053e0 tx=0x7f1f1400b410 comp rx=0 tx=0).stop 2024-04-23T18:16:12.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 msgr2=0x7f1f24103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:12.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000c00 tx=0x7f1f200047b0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 shutdown_connections 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1f100a3cd0 0x7f1f100a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f2410f4f0 0x7f1f241046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f2406a750 0x7f1f24103c20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 --2- 172.21.15.26:0/229088780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2406b070 0x7f1f24104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 >> 172.21.15.26:0/229088780 conn(0x7f1f240788f0 msgr2=0x7f1f241109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 shutdown_connections 2024-04-23T18:16:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:12.470+0000 7f1f2be12700 1 -- 172.21.15.26:0/229088780 wait complete. 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:16:12.659 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:16:12.660 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-23T18:16:12.660 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:16:12.660 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:12 smithi131 ceph-mon[33041]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:16:12.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-23T18:16:12.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:16:12.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:12 smithi110 ceph-mon[32585]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:12.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:16:12.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:12 smithi026 ceph-mon[29364]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:13.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:13 smithi026 ceph-mon[29364]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:16:13.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/229088780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:16:13.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:13 smithi110 ceph-mon[32585]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:16:13.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/229088780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:16:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:13 smithi131 ceph-mon[33041]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:16:13.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/229088780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:16:14.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037795 got 257698037802 for osd.8 2024-04-23T18:16:14.084 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:14.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:14 smithi026 ceph-mon[29364]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:14.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:14 smithi110 ceph-mon[32585]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:14.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:14 smithi131 ceph-mon[33041]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- 172.21.15.26:0/632967190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 msgr2=0x7f705c10efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 --2- 172.21.15.26:0/632967190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c10efb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7044009990 tx=0x7f7044009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- 172.21.15.26:0/632967190 shutdown_connections 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 --2- 172.21.15.26:0/632967190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c10f4f0 0x7f705c1118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 --2- 172.21.15.26:0/632967190 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c06a750 0x7f705c06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 --2- 172.21.15.26:0/632967190 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c10efb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- 172.21.15.26:0/632967190 >> 172.21.15.26:0/632967190 conn(0x7f705c0788f0 msgr2=0x7f705c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- 172.21.15.26:0/632967190 shutdown_connections 2024-04-23T18:16:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- 172.21.15.26:0/632967190 wait complete. 2024-04-23T18:16:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 Processor -- start 2024-04-23T18:16:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.117+0000 7f7062699700 1 -- start start 2024-04-23T18:16:15.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 0x7f705c19b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 0x7f705c19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f705c114e80 con 0x7f705c06b070 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f705c114d00 con 0x7f705c06a750 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7062699700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f705c115000 con 0x7f705c10f4f0 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59118/0 (socket says 172.21.15.26:59118) 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 -- 172.21.15.26:0/4257991048 learned_addr learned my addr 172.21.15.26:0/4257991048 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7060c36700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 0x7f705c19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705bfff700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 0x7f705c19b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 -- 172.21.15.26:0/4257991048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 msgr2=0x7f705c19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 0x7f705c19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 -- 172.21.15.26:0/4257991048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 msgr2=0x7f705c19b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 0x7f705c19b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 -- 172.21.15.26:0/4257991048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705c1a0420 con 0x7f705c06b070 2024-04-23T18:16:15.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f7060c36700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 0x7f705c19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705bfff700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 0x7f705c19b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.118+0000 7f705b7fe700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f70440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.119+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044027070 con 0x7f705c06b070 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.119+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705c1a06b0 con 0x7f705c06b070 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.119+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7044003fa0 con 0x7f705c06b070 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.119+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044022db0 con 0x7f705c06b070 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.119+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705c1a7f90 con 0x7f705c06b070 2024-04-23T18:16:15.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.120+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705c069730 con 0x7f705c06b070 2024-04-23T18:16:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.121+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f704402a070 con 0x7f705c06b070 2024-04-23T18:16:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.121+0000 7f70597fa700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 0x7f70480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.122+0000 7f705bfff700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 0x7f70480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.122+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f70440d1400 con 0x7f705c06b070 2024-04-23T18:16:15.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.122+0000 7f705bfff700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 0x7f70480a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f704c0099d0 tx=0x7f704c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:15.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.126+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f704405eff0 con 0x7f705c06b070 2024-04-23T18:16:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.336+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f705c02df80 con 0x7f705c06b070 2024-04-23T18:16:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.336+0000 7f70597fa700 1 -- 172.21.15.26:0/4257991048 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7044013070 con 0x7f705c06b070 2024-04-23T18:16:15.330 INFO:teuthology.orchestra.run.smithi026.stdout:107374182467 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.338+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 msgr2=0x7f70480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.338+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 0x7f70480a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f704c0099d0 tx=0x7f704c008040 comp rx=0 tx=0).stop 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.338+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 msgr2=0x7f705c19b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.338+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f70440047b0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 shutdown_connections 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f70480a3cd0 0x7f70480a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f705c10f4f0 0x7f705c19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f705c06a750 0x7f705c19b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 --2- 172.21.15.26:0/4257991048 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f705c06b070 0x7f705c19b950 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 >> 172.21.15.26:0/4257991048 conn(0x7f705c0788f0 msgr2=0x7f705c110a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 shutdown_connections 2024-04-23T18:16:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.339+0000 7f7062699700 1 -- 172.21.15.26:0/4257991048 wait complete. 2024-04-23T18:16:15.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920828 for osd.4 2024-04-23T18:16:15.943 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:16.251 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724595 got 167503724600 for osd.5 2024-04-23T18:16:16.251 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:16.602 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:16 smithi026 ceph-mon[29364]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:16.602 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4257991048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:16:16.602 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/724913632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:16:16.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:16 smithi110 ceph-mon[32585]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:16.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4257991048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:16:16.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/724913632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:16:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:16 smithi131 ceph-mon[33041]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 77 B/s, 0 objects/s recovering 2024-04-23T18:16:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4257991048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:16:16.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/724913632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.647+0000 7fab8401e700 1 -- 172.21.15.26:0/1680087125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c06b1e0 msgr2=0x7fab7c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.647+0000 7fab8401e700 1 --2- 172.21.15.26:0/1680087125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c06b1e0 0x7fab7c06b660 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0099c0 tx=0x7fab6c009cd0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.647+0000 7fab8401e700 1 -- 172.21.15.26:0/1680087125 shutdown_connections 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.647+0000 7fab8401e700 1 --2- 172.21.15.26:0/1680087125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06bba0 0x7fab7c10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 --2- 172.21.15.26:0/1680087125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c108a20 0x7fab7c108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 --2- 172.21.15.26:0/1680087125 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c06b1e0 0x7fab7c06b660 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- 172.21.15.26:0/1680087125 >> 172.21.15.26:0/1680087125 conn(0x7fab7c0779b0 msgr2=0x7fab7c077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- 172.21.15.26:0/1680087125 shutdown_connections 2024-04-23T18:16:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- 172.21.15.26:0/1680087125 wait complete. 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 Processor -- start 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- start start 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06b1e0 0x7fab7c101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c06bba0 0x7fab7c101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112bf0 con 0x7fab7c108a20 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112a70 con 0x7fab7c06bba0 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.648+0000 7fab8401e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112d70 con 0x7fab7c06b1e0 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59136/0 (socket says 172.21.15.26:59136) 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 -- 172.21.15.26:0/724913632 learned_addr learned my addr 172.21.15.26:0/724913632 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab815b9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c06bba0 0x7fab7c101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab81dba700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06b1e0 0x7fab7c101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 -- 172.21.15.26:0/724913632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06b1e0 msgr2=0x7fab7c101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06b1e0 0x7fab7c101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 -- 172.21.15.26:0/724913632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c06bba0 msgr2=0x7fab7c101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c06bba0 0x7fab7c101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 -- 172.21.15.26:0/724913632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c1a7690 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab825bb700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fab7400ada0 tx=0x7fab7400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74017020 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab7400ce30 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c1a7980 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74005350 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.649+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c1a7e40 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.650+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c1041a0 con 0x7fab7c108a20 2024-04-23T18:16:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.652+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fab7400c980 con 0x7fab7c108a20 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stdout:313532612639 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.652+0000 7fab72ffd700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 0x7fab680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.653+0000 7fab81dba700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 0x7fab680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.653+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fab740d0310 con 0x7fab7c108a20 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.653+0000 7fab81dba700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 0x7fab680a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab78005950 tx=0x7fab780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.656+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab74021020 con 0x7fab7c108a20 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.863+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fab7c02e140 con 0x7fab7c108a20 2024-04-23T18:16:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.864+0000 7fab72ffd700 1 -- 172.21.15.26:0/724913632 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fab7401a310 con 0x7fab7c108a20 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 msgr2=0x7fab680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 0x7fab680a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab78005950 tx=0x7fab780058e0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 msgr2=0x7fab7c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fab7400ada0 tx=0x7fab7400c330 comp rx=0 tx=0).stop 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 shutdown_connections 2024-04-23T18:16:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fab680a3cd0 0x7fab680a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fab7c108a20 0x7fab7c1024c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fab7c06b1e0 0x7fab7c101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 --2- 172.21.15.26:0/724913632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab7c06bba0 0x7fab7c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 >> 172.21.15.26:0/724913632 conn(0x7fab7c0779b0 msgr2=0x7fab7c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 shutdown_connections 2024-04-23T18:16:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:15.866+0000 7fab8401e700 1 -- 172.21.15.26:0/724913632 wait complete. 2024-04-23T18:16:17.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612634 got 313532612639 for osd.10 2024-04-23T18:16:17.339 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:18.262 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182460 got 107374182467 for osd.3 2024-04-23T18:16:18.262 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:18.555 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:18 smithi026 ceph-mon[29364]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:18.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:18 smithi110 ceph-mon[32585]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:18.741 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:18 smithi131 ceph-mon[33041]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 -- 172.21.15.26:0/1664514567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc101940 msgr2=0x7f6dfc101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 --2- 172.21.15.26:0/1664514567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc101940 0x7f6dfc101dc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009990 tx=0x7f6dec009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 -- 172.21.15.26:0/1664514567 shutdown_connections 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 --2- 172.21.15.26:0/1664514567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc102300 0x7f6dfc10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 --2- 172.21.15.26:0/1664514567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 0x7f6dfc107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 --2- 172.21.15.26:0/1664514567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc101940 0x7f6dfc101dc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 -- 172.21.15.26:0/1664514567 >> 172.21.15.26:0/1664514567 conn(0x7f6dfc0788f0 msgr2=0x7f6dfc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 -- 172.21.15.26:0/1664514567 shutdown_connections 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.971+0000 7f6e0418d700 1 -- 172.21.15.26:0/1664514567 wait complete. 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 Processor -- start 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 -- start start 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc101940 0x7f6dfc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 0x7f6dfc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc111b00 con 0x7f6dfc102300 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc111980 con 0x7f6dfc101940 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0418d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc111c80 con 0x7f6dfc107990 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e01728700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e01728700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42178/0 (socket says 172.21.15.26:42178) 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e01728700 1 -- 172.21.15.26:0/3021775713 learned_addr learned my addr 172.21.15.26:0/3021775713 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e01f29700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc101940 0x7f6dfc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.972+0000 7f6e0272a700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 0x7f6dfc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 -- 172.21.15.26:0/3021775713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 msgr2=0x7f6dfc19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 0x7f6dfc19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 -- 172.21.15.26:0/3021775713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc101940 msgr2=0x7f6dfc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc101940 0x7f6dfc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 -- 172.21.15.26:0/3021775713 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc19d020 con 0x7f6dfc102300 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e01728700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00b5c0 tx=0x7f6dec0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec027070 con 0x7f6dfc102300 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dec003fb0 con 0x7f6dfc102300 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dfc19d2b0 con 0x7f6dfc102300 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec022ce0 con 0x7f6dfc102300 2024-04-23T18:16:18.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.973+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dfc1a4b90 con 0x7f6dfc102300 2024-04-23T18:16:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.974+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dfc109eb0 con 0x7f6dfc102300 2024-04-23T18:16:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.975+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6dec02a070 con 0x7f6dfc102300 2024-04-23T18:16:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.976+0000 7f6df2ffd700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 0x7f6de80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.976+0000 7f6e01f29700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 0x7f6de80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.976+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6dec0d1b90 con 0x7f6dfc102300 2024-04-23T18:16:18.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.976+0000 7f6e01f29700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 0x7f6de80a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6df80045b0 tx=0x7f6df800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:18.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:18.979+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6dec031070 con 0x7f6dfc102300 2024-04-23T18:16:19.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411398 for osd.2 2024-04-23T18:16:19.003 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:19.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.205+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6dfc199370 con 0x7f6dfc102300 2024-04-23T18:16:19.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.206+0000 7f6df2ffd700 1 -- 172.21.15.26:0/3021775713 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6dec013070 con 0x7f6dfc102300 2024-04-23T18:16:19.200 INFO:teuthology.orchestra.run.smithi026.stdout:223338299439 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 msgr2=0x7f6de80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 0x7f6de80a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6df80045b0 tx=0x7f6df800b3c0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 msgr2=0x7f6dfc198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00b5c0 tx=0x7f6dec0058e0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 shutdown_connections 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f6de80a3cd0 0x7f6de80a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6dfc107990 0x7f6dfc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dfc101940 0x7f6dfc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 --2- 172.21.15.26:0/3021775713 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6dfc102300 0x7f6dfc198550 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 >> 172.21.15.26:0/3021775713 conn(0x7f6dfc0788f0 msgr2=0x7f6dfc0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 shutdown_connections 2024-04-23T18:16:19.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:19.208+0000 7f6e0418d700 1 -- 172.21.15.26:0/3021775713 wait complete. 2024-04-23T18:16:19.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:19 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3021775713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:16:19.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:19 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3021775713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:16:19.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:19 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3021775713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:16:20.616 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:20 smithi026 ceph-mon[29364]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.622+0000 7fd887a5a700 1 -- 172.21.15.26:0/2250172208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880102a90 msgr2=0x7fd880102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.622+0000 7fd887a5a700 1 --2- 172.21.15.26:0/2250172208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880102a90 0x7fd880102f10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd870009990 tx=0x7fd870009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 -- 172.21.15.26:0/2250172208 shutdown_connections 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 --2- 172.21.15.26:0/2250172208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 0x7fd88010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 --2- 172.21.15.26:0/2250172208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880108a50 0x7fd880108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 --2- 172.21.15.26:0/2250172208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880102a90 0x7fd880102f10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 -- 172.21.15.26:0/2250172208 >> 172.21.15.26:0/2250172208 conn(0x7fd880079ba0 msgr2=0x7fd880079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 -- 172.21.15.26:0/2250172208 shutdown_connections 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 -- 172.21.15.26:0/2250172208 wait complete. 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 Processor -- start 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.623+0000 7fd887a5a700 1 -- start start 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880102a90 0x7fd880074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 0x7fd880075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112c80 con 0x7fd880108a50 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112b00 con 0x7fd880103450 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd887a5a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112e00 con 0x7fd880102a90 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd8857f6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880102a90 0x7fd880074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42190/0 (socket says 172.21.15.26:42190) 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 -- 172.21.15.26:0/4098452746 learned_addr learned my addr 172.21.15.26:0/4098452746 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd884ff5700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 0x7fd880075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 -- 172.21.15.26:0/4098452746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880102a90 msgr2=0x7fd880074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880102a90 0x7fd880074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 -- 172.21.15.26:0/4098452746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 msgr2=0x7fd880075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 0x7fd880075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 -- 172.21.15.26:0/4098452746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8801a7700 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd885ff7700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd87800b970 tx=0x7fd87800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.624+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878019070 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.625+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87800cde0 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.625+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801a79f0 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.625+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878007370 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.625+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801a7f80 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.626+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd880069730 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.627+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd87801c070 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.627+0000 7fd8767fc700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 0x7fd86c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.627+0000 7fd8857f6700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 0x7fd86c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.627+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd8780cff90 con 0x7fd880108a50 2024-04-23T18:16:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.628+0000 7fd8857f6700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 0x7fd86c0a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd87c004630 tx=0x7fd87c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.631+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd878023070 con 0x7fd880108a50 2024-04-23T18:16:20.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:20 smithi110 ceph-mon[32585]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:20.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:20 smithi131 ceph-mon[33041]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:20.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.844+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fd88002df80 con 0x7fd880108a50 2024-04-23T18:16:20.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.845+0000 7fd8767fc700 1 -- 172.21.15.26:0/4098452746 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd878096e20 con 0x7fd880108a50 2024-04-23T18:16:20.838 INFO:teuthology.orchestra.run.smithi026.stdout:292057776166 2024-04-23T18:16:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 msgr2=0x7fd86c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 0x7fd86c0a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd87c004630 tx=0x7fd87c00b3c0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 msgr2=0x7fd8800757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd87800b970 tx=0x7fd87800bc80 comp rx=0 tx=0).stop 2024-04-23T18:16:20.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 shutdown_connections 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fd86c0a3d50 0x7fd86c0a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd880108a50 0x7fd8800757c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd880102a90 0x7fd880074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 --2- 172.21.15.26:0/4098452746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd880103450 0x7fd880075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 >> 172.21.15.26:0/4098452746 conn(0x7fd880079ba0 msgr2=0x7fd880101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 shutdown_connections 2024-04-23T18:16:20.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.847+0000 7fd887a5a700 1 -- 172.21.15.26:0/4098452746 wait complete. 2024-04-23T18:16:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 -- 172.21.15.26:0/2194486371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 msgr2=0x7fdf5410e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 --2- 172.21.15.26:0/2194486371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5410e7e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00b0a0 tx=0x7fdf4c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 -- 172.21.15.26:0/2194486371 shutdown_connections 2024-04-23T18:16:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 --2- 172.21.15.26:0/2194486371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5410e7e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 --2- 172.21.15.26:0/2194486371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54107980 0x7fdf54107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 --2- 172.21.15.26:0/2194486371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54101930 0x7fdf54101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 -- 172.21.15.26:0/2194486371 >> 172.21.15.26:0/2194486371 conn(0x7fdf540788b0 msgr2=0x7fdf54078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 -- 172.21.15.26:0/2194486371 shutdown_connections 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.855+0000 7fdf58a84700 1 -- 172.21.15.26:0/2194486371 wait complete. 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 Processor -- start 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 -- start start 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54101930 0x7fdf54199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54107980 0x7fdf541a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdf54111ad0 con 0x7fdf541022f0 2024-04-23T18:16:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdf54111950 con 0x7fdf54107980 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.856+0000 7fdf58a84700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdf54111c50 con 0x7fdf54101930 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42194/0 (socket says 172.21.15.26:42194) 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 -- 172.21.15.26:0/3937603929 learned_addr learned my addr 172.21.15.26:0/3937603929 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf5259c700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54101930 0x7fdf54199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf52d9d700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54107980 0x7fdf541a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 -- 172.21.15.26:0/3937603929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54101930 msgr2=0x7fdf54199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54101930 0x7fdf54199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 -- 172.21.15.26:0/3937603929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54107980 msgr2=0x7fdf541a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54107980 0x7fdf541a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 -- 172.21.15.26:0/3937603929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf541a54b0 con 0x7fdf541022f0 2024-04-23T18:16:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf51d9b700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d950 tx=0x7fdf3c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c016070 con 0x7fdf541022f0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf541a56e0 con 0x7fdf541022f0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf3c00fd40 con 0x7fdf541022f0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c00e3b0 con 0x7fdf541022f0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.857+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf541a5bf0 con 0x7fdf541022f0 2024-04-23T18:16:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.858+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf54069730 con 0x7fdf541022f0 2024-04-23T18:16:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.860+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf3c019070 con 0x7fdf541022f0 2024-04-23T18:16:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.861+0000 7fdf4b7fe700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 0x7fdf400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.861+0000 7fdf5259c700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 0x7fdf400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.861+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdf3c0cfff0 con 0x7fdf541022f0 2024-04-23T18:16:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.862+0000 7fdf5259c700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 0x7fdf400a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005950 tx=0x7fdf4400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:20.865+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf3c09c840 con 0x7fdf541022f0 2024-04-23T18:16:21.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.074+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdf5402df80 con 0x7fdf541022f0 2024-04-23T18:16:21.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.075+0000 7fdf4b7fe700 1 -- 172.21.15.26:0/3937603929 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdf3c09c140 con 0x7fdf541022f0 2024-04-23T18:16:21.069 INFO:teuthology.orchestra.run.smithi026.stdout:201863462965 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 msgr2=0x7fdf400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 0x7fdf400a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005950 tx=0x7fdf4400a300 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 msgr2=0x7fdf5419a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d950 tx=0x7fdf3c00dc60 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 shutdown_connections 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fdf400a3cd0 0x7fdf400a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf54107980 0x7fdf541a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf54101930 0x7fdf54199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 --2- 172.21.15.26:0/3937603929 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdf541022f0 0x7fdf5419a430 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 >> 172.21.15.26:0/3937603929 conn(0x7fdf540788b0 msgr2=0x7fdf540ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 shutdown_connections 2024-04-23T18:16:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:21.077+0000 7fdf58a84700 1 -- 172.21.15.26:0/3937603929 wait complete. 2024-04-23T18:16:21.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:21 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4098452746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:16:21.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:21 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3937603929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:16:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:21 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4098452746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:16:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:21 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3937603929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:16:21.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:21 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4098452746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:16:21.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:21 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3937603929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:16:21.743 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462965 for osd.6 2024-04-23T18:16:21.743 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:22.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299434 got 223338299439 for osd.7 2024-04-23T18:16:22.224 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:22.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:22 smithi026 ceph-mon[29364]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 68 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:22.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776161 got 292057776166 for osd.9 2024-04-23T18:16:22.644 DEBUG:teuthology.parallel:result is None 2024-04-23T18:16:22.644 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T18:16:22.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:16:22.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:22 smithi110 ceph-mon[32585]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 68 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:22.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:22 smithi131 ceph-mon[33041]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 68 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T18:16:22.925 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:24.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:23 smithi131 ceph-mon[33041]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.291 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:24.291 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.291 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.291 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:24.292 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:24.292 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:23 smithi026 ceph-mon[29364]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:24.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:23 smithi110 ceph-mon[32585]: from='mgr.14152 172.21.15.26:0/4072967866' entity='mgr.x' 2024-04-23T18:16:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.674+0000 7fcb4272c700 1 -- 172.21.15.26:0/2749155348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 msgr2=0x7fcb3c10e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.674+0000 7fcb4272c700 1 --2- 172.21.15.26:0/2749155348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c10e830 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400b0a0 tx=0x7fcb3400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 -- 172.21.15.26:0/2749155348 shutdown_connections 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 --2- 172.21.15.26:0/2749155348 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c10e830 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 --2- 172.21.15.26:0/2749155348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c107960 0x7fcb3c107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 --2- 172.21.15.26:0/2749155348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c101960 0x7fcb3c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 -- 172.21.15.26:0/2749155348 >> 172.21.15.26:0/2749155348 conn(0x7fcb3c0789e0 msgr2=0x7fcb3c078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 -- 172.21.15.26:0/2749155348 shutdown_connections 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 -- 172.21.15.26:0/2749155348 wait complete. 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 Processor -- start 2024-04-23T18:16:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.675+0000 7fcb4272c700 1 -- start start 2024-04-23T18:16:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c101960 0x7fcb3c0fe990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c107960 0x7fcb3c0ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c111bf0 con 0x7fcb3c102320 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c111a70 con 0x7fcb3c107960 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb4272c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c111d70 con 0x7fcb3c101960 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42210/0 (socket says 172.21.15.26:42210) 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 -- 172.21.15.26:0/3669432798 learned_addr learned my addr 172.21.15.26:0/3669432798 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3bfff700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c101960 0x7fcb3c0fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb40cc9700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c107960 0x7fcb3c0ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 -- 172.21.15.26:0/3669432798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c101960 msgr2=0x7fcb3c0fe990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c101960 0x7fcb3c0fe990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 -- 172.21.15.26:0/3669432798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c107960 msgr2=0x7fcb3c0ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c107960 0x7fcb3c0ff410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 -- 172.21.15.26:0/3669432798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c1a7640 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb3b7fe700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000d980 tx=0x7fcb3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.676+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30016070 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.677+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb30004830 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.677+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3c1a7930 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.677+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30005540 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.677+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb3c1a7ec0 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.677+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3c069730 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.679+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcb30004220 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.680+0000 7fcb397fa700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 0x7fcb240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.680+0000 7fcb3bfff700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 0x7fcb240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.680+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcb300d07e0 con 0x7fcb3c102320 2024-04-23T18:16:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.680+0000 7fcb3bfff700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 0x7fcb240a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005950 tx=0x7fcb2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.684+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb3001f020 con 0x7fcb3c102320 2024-04-23T18:16:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.884+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb3c062e80 con 0x7fcb240a3d50 2024-04-23T18:16:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.892+0000 7fcb397fa700 1 -- 172.21.15.26:0/3669432798 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258710 (secure 0 0 0) 0x7fcb3c062e80 con 0x7fcb240a3d50 2024-04-23T18:16:24.889 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 msgr2=0x7fcb240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 0x7fcb240a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005950 tx=0x7fcb2c00b410 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 msgr2=0x7fcb3c0feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000d980 tx=0x7fcb3000dc90 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 shutdown_connections 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcb240a3d50 0x7fcb240a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c107960 0x7fcb3c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcb3c101960 0x7fcb3c0fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 --2- 172.21.15.26:0/3669432798 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb3c102320 0x7fcb3c0feed0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 >> 172.21.15.26:0/3669432798 conn(0x7fcb3c0789e0 msgr2=0x7fcb3c06b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 shutdown_connections 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:24.895+0000 7fcb4272c700 1 -- 172.21.15.26:0/3669432798 wait complete. 2024-04-23T18:16:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:16:25.623 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-23T18:16:24.857999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655649,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":581,"num_read_kb":408,"num_write":339,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181392,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399972,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124541852,"statfs":{"total":1151940427776,"available":1151530856448,"internally_reserved":0,"allocated":13627392,"data_stored":7488351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":28,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003197"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301386+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:16:12.301386+0000","last_peered":"2024-04-23T18:16:12.301386+0000","last_clean":"2024-04-23T18:16:12.301386+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:16:12.301386+0000","last_undegraded":"2024-04-23T18:16:12.301386+0000","last_fullsized":"2024-04-23T18:16:12.301386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302779+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:16:12.302779+0000","last_peered":"2024-04-23T18:16:12.302779+0000","last_clean":"2024-04-23T18:16:12.302779+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:16:12.302779+0000","last_undegraded":"2024-04-23T18:16:12.302779+0000","last_fullsized":"2024-04-23T18:16:12.302779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302542+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:16:12.302542+0000","last_peered":"2024-04-23T18:16:12.302542+0000","last_clean":"2024-04-23T18:16:12.302542+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:16:12.302542+0000","last_undegraded":"2024-04-23T18:16:12.302542+0000","last_fullsized":"2024-04-23T18:16:12.302542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301752+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:16:12.301752+0000","last_peered":"2024-04-23T18:16:12.301752+0000","last_clean":"2024-04-23T18:16:12.301752+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:16:12.301752+0000","last_undegraded":"2024-04-23T18:16:12.301752+0000","last_fullsized":"2024-04-23T18:16:12.301752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300906+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:16:12.300906+0000","last_peered":"2024-04-23T18:16:12.300906+0000","last_clean":"2024-04-23T18:16:12.300906+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:16:12.300906+0000","last_undegraded":"2024-04-23T18:16:12.300906+0000","last_fullsized":"2024-04-23T18:16:12.300906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.298749+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:16:12.298749+0000","last_peered":"2024-04-23T18:16:12.298749+0000","last_clean":"2024-04-23T18:16:12.298749+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:16:12.298749+0000","last_undegraded":"2024-04-23T18:16:12.298749+0000","last_fullsized":"2024-04-23T18:16:12.298749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301536+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:16:12.301536+0000","last_peered":"2024-04-23T18:16:12.301536+0000","last_clean":"2024-04-23T18:16:12.301536+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:16:12.301536+0000","last_undegraded":"2024-04-23T18:16:12.301536+0000","last_fullsized":"2024-04-23T18:16:12.301536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300478+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:16:12.300478+0000","last_peered":"2024-04-23T18:16:12.300478+0000","last_clean":"2024-04-23T18:16:12.300478+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:16:12.300478+0000","last_undegraded":"2024-04-23T18:16:12.300478+0000","last_fullsized":"2024-04-23T18:16:12.300478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301082+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:16:12.301082+0000","last_peered":"2024-04-23T18:16:12.301082+0000","last_clean":"2024-04-23T18:16:12.301082+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:16:12.301082+0000","last_undegraded":"2024-04-23T18:16:12.301082+0000","last_fullsized":"2024-04-23T18:16:12.301082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.299182+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:16:12.299182+0000","last_peered":"2024-04-23T18:16:12.299182+0000","last_clean":"2024-04-23T18:16:12.299182+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:16:12.299182+0000","last_undegraded":"2024-04-23T18:16:12.299182+0000","last_fullsized":"2024-04-23T18:16:12.299182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301650+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:16:12.301650+0000","last_peered":"2024-04-23T18:16:12.301650+0000","last_clean":"2024-04-23T18:16:12.301650+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:16:12.301650+0000","last_undegraded":"2024-04-23T18:16:12.301650+0000","last_fullsized":"2024-04-23T18:16:12.301650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301670+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:16:12.301670+0000","last_peered":"2024-04-23T18:16:12.301670+0000","last_clean":"2024-04-23T18:16:12.301670+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:16:12.301670+0000","last_undegraded":"2024-04-23T18:16:12.301670+0000","last_fullsized":"2024-04-23T18:16:12.301670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301538+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:16:12.301538+0000","last_peered":"2024-04-23T18:16:12.301538+0000","last_clean":"2024-04-23T18:16:12.301538+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:16:12.301538+0000","last_undegraded":"2024-04-23T18:16:12.301538+0000","last_fullsized":"2024-04-23T18:16:12.301538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301811+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:16:12.301811+0000","last_peered":"2024-04-23T18:16:12.301811+0000","last_clean":"2024-04-23T18:16:12.301811+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:16:12.301811+0000","last_undegraded":"2024-04-23T18:16:12.301811+0000","last_fullsized":"2024-04-23T18:16:12.301811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302097+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:16:12.302097+0000","last_peered":"2024-04-23T18:16:12.302097+0000","last_clean":"2024-04-23T18:16:12.302097+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:16:12.302097+0000","last_undegraded":"2024-04-23T18:16:12.302097+0000","last_fullsized":"2024-04-23T18:16:12.302097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300531+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:16:12.300531+0000","last_peered":"2024-04-23T18:16:12.300531+0000","last_clean":"2024-04-23T18:16:12.300531+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:16:12.300531+0000","last_undegraded":"2024-04-23T18:16:12.300531+0000","last_fullsized":"2024-04-23T18:16:12.300531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301572+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:16:12.301572+0000","last_peered":"2024-04-23T18:16:12.301572+0000","last_clean":"2024-04-23T18:16:12.301572+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:16:12.301572+0000","last_undegraded":"2024-04-23T18:16:12.301572+0000","last_fullsized":"2024-04-23T18:16:12.301572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297724+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:16:12.297724+0000","last_peered":"2024-04-23T18:16:12.297724+0000","last_clean":"2024-04-23T18:16:12.297724+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:16:12.297724+0000","last_undegraded":"2024-04-23T18:16:12.297724+0000","last_fullsized":"2024-04-23T18:16:12.297724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302289+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:16:12.302289+0000","last_peered":"2024-04-23T18:16:12.302289+0000","last_clean":"2024-04-23T18:16:12.302289+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:16:12.302289+0000","last_undegraded":"2024-04-23T18:16:12.302289+0000","last_fullsized":"2024-04-23T18:16:12.302289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297643+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:16:12.297643+0000","last_peered":"2024-04-23T18:16:12.297643+0000","last_clean":"2024-04-23T18:16:12.297643+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:16:12.297643+0000","last_undegraded":"2024-04-23T18:16:12.297643+0000","last_fullsized":"2024-04-23T18:16:12.297643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301485+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:16:12.301485+0000","last_peered":"2024-04-23T18:16:12.301485+0000","last_clean":"2024-04-23T18:16:12.301485+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:16:12.301485+0000","last_undegraded":"2024-04-23T18:16:12.301485+0000","last_fullsized":"2024-04-23T18:16:12.301485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301775+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:16:12.301775+0000","last_peered":"2024-04-23T18:16:12.301775+0000","last_clean":"2024-04-23T18:16:12.301775+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:16:12.301775+0000","last_undegraded":"2024-04-23T18:16:12.301775+0000","last_fullsized":"2024-04-23T18:16:12.301775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303016+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:16:12.303016+0000","last_peered":"2024-04-23T18:16:12.303016+0000","last_clean":"2024-04-23T18:16:12.303016+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:16:12.303016+0000","last_undegraded":"2024-04-23T18:16:12.303016+0000","last_fullsized":"2024-04-23T18:16:12.303016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304495+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:16:12.304495+0000","last_peered":"2024-04-23T18:16:12.304495+0000","last_clean":"2024-04-23T18:16:12.304495+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:16:12.304495+0000","last_undegraded":"2024-04-23T18:16:12.304495+0000","last_fullsized":"2024-04-23T18:16:12.304495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300889+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:16:12.300889+0000","last_peered":"2024-04-23T18:16:12.300889+0000","last_clean":"2024-04-23T18:16:12.300889+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:16:12.300889+0000","last_undegraded":"2024-04-23T18:16:12.300889+0000","last_fullsized":"2024-04-23T18:16:12.300889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301281+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:16:12.301281+0000","last_peered":"2024-04-23T18:16:12.301281+0000","last_clean":"2024-04-23T18:16:12.301281+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:16:12.301281+0000","last_undegraded":"2024-04-23T18:16:12.301281+0000","last_fullsized":"2024-04-23T18:16:12.301281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302137+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:16:12.302137+0000","last_peered":"2024-04-23T18:16:12.302137+0000","last_clean":"2024-04-23T18:16:12.302137+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:16:12.302137+0000","last_undegraded":"2024-04-23T18:16:12.302137+0000","last_fullsized":"2024-04-23T18:16:12.302137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:21.318849+0000","last_change":"2024-04-23T18:16:12.309950+0000","last_active":"2024-04-23T18:16:21.318849+0000","last_peered":"2024-04-23T18:16:21.318849+0000","last_clean":"2024-04-23T18:16:21.318849+0000","last_became_active":"2024-04-23T18:16:12.305208+0000","last_became_peered":"2024-04-23T18:16:12.305208+0000","last_unstale":"2024-04-23T18:16:21.318849+0000","last_undegraded":"2024-04-23T18:16:21.318849+0000","last_fullsized":"2024-04-23T18:16:21.318849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:02.996009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300811+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:16:12.300811+0000","last_peered":"2024-04-23T18:16:12.300811+0000","last_clean":"2024-04-23T18:16:12.300811+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:16:12.300811+0000","last_undegraded":"2024-04-23T18:16:12.300811+0000","last_fullsized":"2024-04-23T18:16:12.300811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.298721+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:16:12.298721+0000","last_peered":"2024-04-23T18:16:12.298721+0000","last_clean":"2024-04-23T18:16:12.298721+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:16:12.298721+0000","last_undegraded":"2024-04-23T18:16:12.298721+0000","last_fullsized":"2024-04-23T18:16:12.298721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303154+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:16:12.303154+0000","last_peered":"2024-04-23T18:16:12.303154+0000","last_clean":"2024-04-23T18:16:12.303154+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:16:12.303154+0000","last_undegraded":"2024-04-23T18:16:12.303154+0000","last_fullsized":"2024-04-23T18:16:12.303154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300600+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:16:12.300600+0000","last_peered":"2024-04-23T18:16:12.300600+0000","last_clean":"2024-04-23T18:16:12.300600+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:16:12.300600+0000","last_undegraded":"2024-04-23T18:16:12.300600+0000","last_fullsized":"2024-04-23T18:16:12.300600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300513+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:16:12.300513+0000","last_peered":"2024-04-23T18:16:12.300513+0000","last_clean":"2024-04-23T18:16:12.300513+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:16:12.300513+0000","last_undegraded":"2024-04-23T18:16:12.300513+0000","last_fullsized":"2024-04-23T18:16:12.300513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301490+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:16:12.301490+0000","last_peered":"2024-04-23T18:16:12.301490+0000","last_clean":"2024-04-23T18:16:12.301490+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:16:12.301490+0000","last_undegraded":"2024-04-23T18:16:12.301490+0000","last_fullsized":"2024-04-23T18:16:12.301490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297531+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:16:12.297531+0000","last_peered":"2024-04-23T18:16:12.297531+0000","last_clean":"2024-04-23T18:16:12.297531+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:16:12.297531+0000","last_undegraded":"2024-04-23T18:16:12.297531+0000","last_fullsized":"2024-04-23T18:16:12.297531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:19.315705+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:16:19.315705+0000","last_peered":"2024-04-23T18:16:19.315705+0000","last_clean":"2024-04-23T18:16:19.315705+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:16:19.315705+0000","last_undegraded":"2024-04-23T18:16:19.315705+0000","last_fullsized":"2024-04-23T18:16:19.315705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302279+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:16:12.302279+0000","last_peered":"2024-04-23T18:16:12.302279+0000","last_clean":"2024-04-23T18:16:12.302279+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:16:12.302279+0000","last_undegraded":"2024-04-23T18:16:12.302279+0000","last_fullsized":"2024-04-23T18:16:12.302279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301850+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:16:12.301850+0000","last_peered":"2024-04-23T18:16:12.301850+0000","last_clean":"2024-04-23T18:16:12.301850+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:16:12.301850+0000","last_undegraded":"2024-04-23T18:16:12.301850+0000","last_fullsized":"2024-04-23T18:16:12.301850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.306548+0000","last_change":"2024-04-23T18:16:12.306548+0000","last_active":"2024-04-23T18:16:12.306548+0000","last_peered":"2024-04-23T18:16:12.306548+0000","last_clean":"2024-04-23T18:16:12.306548+0000","last_became_active":"2024-04-23T18:16:12.306227+0000","last_became_peered":"2024-04-23T18:16:12.306227+0000","last_unstale":"2024-04-23T18:16:12.306548+0000","last_undegraded":"2024-04-23T18:16:12.306548+0000","last_fullsized":"2024-04-23T18:16:12.306548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:49.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302291+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:16:12.302291+0000","last_peered":"2024-04-23T18:16:12.302291+0000","last_clean":"2024-04-23T18:16:12.302291+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:16:12.302291+0000","last_undegraded":"2024-04-23T18:16:12.302291+0000","last_fullsized":"2024-04-23T18:16:12.302291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304182+0000","last_change":"2024-04-23T18:16:12.304182+0000","last_active":"2024-04-23T18:16:12.304182+0000","last_peered":"2024-04-23T18:16:12.304182+0000","last_clean":"2024-04-23T18:16:12.304182+0000","last_became_active":"2024-04-23T18:16:12.303708+0000","last_became_peered":"2024-04-23T18:16:12.303708+0000","last_unstale":"2024-04-23T18:16:12.304182+0000","last_undegraded":"2024-04-23T18:16:12.304182+0000","last_fullsized":"2024-04-23T18:16:12.304182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:32.224284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'28","reported_seq":193,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:20.317111+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:16:20.317111+0000","last_peered":"2024-04-23T18:16:20.317111+0000","last_clean":"2024-04-23T18:16:20.317111+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:16:20.317111+0000","last_undegraded":"2024-04-23T18:16:20.317111+0000","last_fullsized":"2024-04-23T18:16:20.317111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":132,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304348+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:16:12.304348+0000","last_peered":"2024-04-23T18:16:12.304348+0000","last_clean":"2024-04-23T18:16:12.304348+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:16:12.304348+0000","last_undegraded":"2024-04-23T18:16:12.304348+0000","last_fullsized":"2024-04-23T18:16:12.304348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.314870+0000","last_change":"2024-04-23T18:16:12.314402+0000","last_active":"2024-04-23T18:16:12.314870+0000","last_peered":"2024-04-23T18:16:12.314870+0000","last_clean":"2024-04-23T18:16:12.314870+0000","last_became_active":"2024-04-23T18:16:12.305588+0000","last_became_peered":"2024-04-23T18:16:12.305588+0000","last_unstale":"2024-04-23T18:16:12.314870+0000","last_undegraded":"2024-04-23T18:16:12.314870+0000","last_fullsized":"2024-04-23T18:16:12.314870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:15.413380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301245+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:16:12.301245+0000","last_peered":"2024-04-23T18:16:12.301245+0000","last_clean":"2024-04-23T18:16:12.301245+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:16:12.301245+0000","last_undegraded":"2024-04-23T18:16:12.301245+0000","last_fullsized":"2024-04-23T18:16:12.301245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301274+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:16:12.301274+0000","last_peered":"2024-04-23T18:16:12.301274+0000","last_clean":"2024-04-23T18:16:12.301274+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:16:12.301274+0000","last_undegraded":"2024-04-23T18:16:12.301274+0000","last_fullsized":"2024-04-23T18:16:12.301274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302760+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:16:12.302760+0000","last_peered":"2024-04-23T18:16:12.302760+0000","last_clean":"2024-04-23T18:16:12.302760+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:16:12.302760+0000","last_undegraded":"2024-04-23T18:16:12.302760+0000","last_fullsized":"2024-04-23T18:16:12.302760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":99,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:22.776995+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:16:22.776995+0000","last_peered":"2024-04-23T18:16:22.776995+0000","last_clean":"2024-04-23T18:16:22.776995+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:16:22.776995+0000","last_undegraded":"2024-04-23T18:16:22.776995+0000","last_fullsized":"2024-04-23T18:16:22.776995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301219+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:16:12.301219+0000","last_peered":"2024-04-23T18:16:12.301219+0000","last_clean":"2024-04-23T18:16:12.301219+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:16:12.301219+0000","last_undegraded":"2024-04-23T18:16:12.301219+0000","last_fullsized":"2024-04-23T18:16:12.301219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300810+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:16:12.300810+0000","last_peered":"2024-04-23T18:16:12.300810+0000","last_clean":"2024-04-23T18:16:12.300810+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:16:12.300810+0000","last_undegraded":"2024-04-23T18:16:12.300810+0000","last_fullsized":"2024-04-23T18:16:12.300810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300937+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:16:12.300937+0000","last_peered":"2024-04-23T18:16:12.300937+0000","last_clean":"2024-04-23T18:16:12.300937+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:16:12.300937+0000","last_undegraded":"2024-04-23T18:16:12.300937+0000","last_fullsized":"2024-04-23T18:16:12.300937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304603+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:16:12.304603+0000","last_peered":"2024-04-23T18:16:12.304603+0000","last_clean":"2024-04-23T18:16:12.304603+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:16:12.304603+0000","last_undegraded":"2024-04-23T18:16:12.304603+0000","last_fullsized":"2024-04-23T18:16:12.304603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304948+0000","last_change":"2024-04-23T18:16:12.304948+0000","last_active":"2024-04-23T18:16:12.304948+0000","last_peered":"2024-04-23T18:16:12.304948+0000","last_clean":"2024-04-23T18:16:12.304948+0000","last_became_active":"2024-04-23T18:16:12.304541+0000","last_became_peered":"2024-04-23T18:16:12.304541+0000","last_unstale":"2024-04-23T18:16:12.304948+0000","last_undegraded":"2024-04-23T18:16:12.304948+0000","last_fullsized":"2024-04-23T18:16:12.304948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:47.148967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297738+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:16:12.297738+0000","last_peered":"2024-04-23T18:16:12.297738+0000","last_clean":"2024-04-23T18:16:12.297738+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:16:12.297738+0000","last_undegraded":"2024-04-23T18:16:12.297738+0000","last_fullsized":"2024-04-23T18:16:12.297738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.299155+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:16:12.299155+0000","last_peered":"2024-04-23T18:16:12.299155+0000","last_clean":"2024-04-23T18:16:12.299155+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:16:12.299155+0000","last_undegraded":"2024-04-23T18:16:12.299155+0000","last_fullsized":"2024-04-23T18:16:12.299155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301910+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:16:12.301910+0000","last_peered":"2024-04-23T18:16:12.301910+0000","last_clean":"2024-04-23T18:16:12.301910+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:16:12.301910+0000","last_undegraded":"2024-04-23T18:16:12.301910+0000","last_fullsized":"2024-04-23T18:16:12.301910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304513+0000","last_change":"2024-04-23T18:16:12.304513+0000","last_active":"2024-04-23T18:16:12.304513+0000","last_peered":"2024-04-23T18:16:12.304513+0000","last_clean":"2024-04-23T18:16:12.304513+0000","last_became_active":"2024-04-23T18:16:12.304069+0000","last_became_peered":"2024-04-23T18:16:12.304069+0000","last_unstale":"2024-04-23T18:16:12.304513+0000","last_undegraded":"2024-04-23T18:16:12.304513+0000","last_fullsized":"2024-04-23T18:16:12.304513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:58.834388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301712+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:16:12.301712+0000","last_peered":"2024-04-23T18:16:12.301712+0000","last_clean":"2024-04-23T18:16:12.301712+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:16:12.301712+0000","last_undegraded":"2024-04-23T18:16:12.301712+0000","last_fullsized":"2024-04-23T18:16:12.301712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:20.772445+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:16:20.772445+0000","last_peered":"2024-04-23T18:16:20.772445+0000","last_clean":"2024-04-23T18:16:20.772445+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:16:20.772445+0000","last_undegraded":"2024-04-23T18:16:20.772445+0000","last_fullsized":"2024-04-23T18:16:20.772445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301942+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:16:12.301942+0000","last_peered":"2024-04-23T18:16:12.301942+0000","last_clean":"2024-04-23T18:16:12.301942+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:16:12.301942+0000","last_undegraded":"2024-04-23T18:16:12.301942+0000","last_fullsized":"2024-04-23T18:16:12.301942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301317+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:16:12.301317+0000","last_peered":"2024-04-23T18:16:12.301317+0000","last_clean":"2024-04-23T18:16:12.301317+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:16:12.301317+0000","last_undegraded":"2024-04-23T18:16:12.301317+0000","last_fullsized":"2024-04-23T18:16:12.301317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301459+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:16:12.301459+0000","last_peered":"2024-04-23T18:16:12.301459+0000","last_clean":"2024-04-23T18:16:12.301459+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:16:12.301459+0000","last_undegraded":"2024-04-23T18:16:12.301459+0000","last_fullsized":"2024-04-23T18:16:12.301459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300375+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:16:12.300375+0000","last_peered":"2024-04-23T18:16:12.300375+0000","last_clean":"2024-04-23T18:16:12.300375+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:16:12.300375+0000","last_undegraded":"2024-04-23T18:16:12.300375+0000","last_fullsized":"2024-04-23T18:16:12.300375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301684+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:16:12.301684+0000","last_peered":"2024-04-23T18:16:12.301684+0000","last_clean":"2024-04-23T18:16:12.301684+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:16:12.301684+0000","last_undegraded":"2024-04-23T18:16:12.301684+0000","last_fullsized":"2024-04-23T18:16:12.301684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304385+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:16:12.304385+0000","last_peered":"2024-04-23T18:16:12.304385+0000","last_clean":"2024-04-23T18:16:12.304385+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:16:12.304385+0000","last_undegraded":"2024-04-23T18:16:12.304385+0000","last_fullsized":"2024-04-23T18:16:12.304385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301325+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:16:12.301325+0000","last_peered":"2024-04-23T18:16:12.301325+0000","last_clean":"2024-04-23T18:16:12.301325+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:16:12.301325+0000","last_undegraded":"2024-04-23T18:16:12.301325+0000","last_fullsized":"2024-04-23T18:16:12.301325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297756+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:16:12.297756+0000","last_peered":"2024-04-23T18:16:12.297756+0000","last_clean":"2024-04-23T18:16:12.297756+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:16:12.297756+0000","last_undegraded":"2024-04-23T18:16:12.297756+0000","last_fullsized":"2024-04-23T18:16:12.297756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301997+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:16:12.301997+0000","last_peered":"2024-04-23T18:16:12.301997+0000","last_clean":"2024-04-23T18:16:12.301997+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:16:12.301997+0000","last_undegraded":"2024-04-23T18:16:12.301997+0000","last_fullsized":"2024-04-23T18:16:12.301997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301113+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:16:12.301113+0000","last_peered":"2024-04-23T18:16:12.301113+0000","last_clean":"2024-04-23T18:16:12.301113+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:16:12.301113+0000","last_undegraded":"2024-04-23T18:16:12.301113+0000","last_fullsized":"2024-04-23T18:16:12.301113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302832+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:16:12.302832+0000","last_peered":"2024-04-23T18:16:12.302832+0000","last_clean":"2024-04-23T18:16:12.302832+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:16:12.302832+0000","last_undegraded":"2024-04-23T18:16:12.302832+0000","last_fullsized":"2024-04-23T18:16:12.302832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300544+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:16:12.300544+0000","last_peered":"2024-04-23T18:16:12.300544+0000","last_clean":"2024-04-23T18:16:12.300544+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:16:12.300544+0000","last_undegraded":"2024-04-23T18:16:12.300544+0000","last_fullsized":"2024-04-23T18:16:12.300544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300961+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:16:12.300961+0000","last_peered":"2024-04-23T18:16:12.300961+0000","last_clean":"2024-04-23T18:16:12.300961+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:16:12.300961+0000","last_undegraded":"2024-04-23T18:16:12.300961+0000","last_fullsized":"2024-04-23T18:16:12.300961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300731+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:16:12.300731+0000","last_peered":"2024-04-23T18:16:12.300731+0000","last_clean":"2024-04-23T18:16:12.300731+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:16:12.300731+0000","last_undegraded":"2024-04-23T18:16:12.300731+0000","last_fullsized":"2024-04-23T18:16:12.300731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301401+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:16:12.301401+0000","last_peered":"2024-04-23T18:16:12.301401+0000","last_clean":"2024-04-23T18:16:12.301401+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:16:12.301401+0000","last_undegraded":"2024-04-23T18:16:12.301401+0000","last_fullsized":"2024-04-23T18:16:12.301401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301615+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:16:12.301615+0000","last_peered":"2024-04-23T18:16:12.301615+0000","last_clean":"2024-04-23T18:16:12.301615+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:16:12.301615+0000","last_undegraded":"2024-04-23T18:16:12.301615+0000","last_fullsized":"2024-04-23T18:16:12.301615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304844+0000","last_change":"2024-04-23T18:16:12.304138+0000","last_active":"2024-04-23T18:16:12.304844+0000","last_peered":"2024-04-23T18:16:12.304844+0000","last_clean":"2024-04-23T18:16:12.304844+0000","last_became_active":"2024-04-23T18:16:12.303760+0000","last_became_peered":"2024-04-23T18:16:12.303760+0000","last_unstale":"2024-04-23T18:16:12.304844+0000","last_undegraded":"2024-04-23T18:16:12.304844+0000","last_fullsized":"2024-04-23T18:16:12.304844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:10.049511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.318277+0000","last_change":"2024-04-23T18:16:12.317810+0000","last_active":"2024-04-23T18:16:12.318277+0000","last_peered":"2024-04-23T18:16:12.318277+0000","last_clean":"2024-04-23T18:16:12.318277+0000","last_became_active":"2024-04-23T18:16:12.307064+0000","last_became_peered":"2024-04-23T18:16:12.307064+0000","last_unstale":"2024-04-23T18:16:12.318277+0000","last_undegraded":"2024-04-23T18:16:12.318277+0000","last_fullsized":"2024-04-23T18:16:12.318277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:06:41.038382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302024+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:16:12.302024+0000","last_peered":"2024-04-23T18:16:12.302024+0000","last_clean":"2024-04-23T18:16:12.302024+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:16:12.302024+0000","last_undegraded":"2024-04-23T18:16:12.302024+0000","last_fullsized":"2024-04-23T18:16:12.302024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301974+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:16:12.301974+0000","last_peered":"2024-04-23T18:16:12.301974+0000","last_clean":"2024-04-23T18:16:12.301974+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:16:12.301974+0000","last_undegraded":"2024-04-23T18:16:12.301974+0000","last_fullsized":"2024-04-23T18:16:12.301974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301713+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:16:12.301713+0000","last_peered":"2024-04-23T18:16:12.301713+0000","last_clean":"2024-04-23T18:16:12.301713+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:16:12.301713+0000","last_undegraded":"2024-04-23T18:16:12.301713+0000","last_fullsized":"2024-04-23T18:16:12.301713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300940+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:16:12.300940+0000","last_peered":"2024-04-23T18:16:12.300940+0000","last_clean":"2024-04-23T18:16:12.300940+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:16:12.300940+0000","last_undegraded":"2024-04-23T18:16:12.300940+0000","last_fullsized":"2024-04-23T18:16:12.300940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301454+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:16:12.301454+0000","last_peered":"2024-04-23T18:16:12.301454+0000","last_clean":"2024-04-23T18:16:12.301454+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:16:12.301454+0000","last_undegraded":"2024-04-23T18:16:12.301454+0000","last_fullsized":"2024-04-23T18:16:12.301454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301135+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:16:12.301135+0000","last_peered":"2024-04-23T18:16:12.301135+0000","last_clean":"2024-04-23T18:16:12.301135+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:16:12.301135+0000","last_undegraded":"2024-04-23T18:16:12.301135+0000","last_fullsized":"2024-04-23T18:16:12.301135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302422+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:16:12.302422+0000","last_peered":"2024-04-23T18:16:12.302422+0000","last_clean":"2024-04-23T18:16:12.302422+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:16:12.302422+0000","last_undegraded":"2024-04-23T18:16:12.302422+0000","last_fullsized":"2024-04-23T18:16:12.302422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301119+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:16:12.301119+0000","last_peered":"2024-04-23T18:16:12.301119+0000","last_clean":"2024-04-23T18:16:12.301119+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:16:12.301119+0000","last_undegraded":"2024-04-23T18:16:12.301119+0000","last_fullsized":"2024-04-23T18:16:12.301119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301385+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:16:12.301385+0000","last_peered":"2024-04-23T18:16:12.301385+0000","last_clean":"2024-04-23T18:16:12.301385+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:16:12.301385+0000","last_undegraded":"2024-04-23T18:16:12.301385+0000","last_fullsized":"2024-04-23T18:16:12.301385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301516+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:16:12.301516+0000","last_peered":"2024-04-23T18:16:12.301516+0000","last_clean":"2024-04-23T18:16:12.301516+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:16:12.301516+0000","last_undegraded":"2024-04-23T18:16:12.301516+0000","last_fullsized":"2024-04-23T18:16:12.301516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303277+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:16:12.303277+0000","last_peered":"2024-04-23T18:16:12.303277+0000","last_clean":"2024-04-23T18:16:12.303277+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:16:12.303277+0000","last_undegraded":"2024-04-23T18:16:12.303277+0000","last_fullsized":"2024-04-23T18:16:12.303277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301421+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:16:12.301421+0000","last_peered":"2024-04-23T18:16:12.301421+0000","last_clean":"2024-04-23T18:16:12.301421+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:16:12.301421+0000","last_undegraded":"2024-04-23T18:16:12.301421+0000","last_fullsized":"2024-04-23T18:16:12.301421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301491+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:16:12.301491+0000","last_peered":"2024-04-23T18:16:12.301491+0000","last_clean":"2024-04-23T18:16:12.301491+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:16:12.301491+0000","last_undegraded":"2024-04-23T18:16:12.301491+0000","last_fullsized":"2024-04-23T18:16:12.301491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302688+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:16:12.302688+0000","last_peered":"2024-04-23T18:16:12.302688+0000","last_clean":"2024-04-23T18:16:12.302688+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:16:12.302688+0000","last_undegraded":"2024-04-23T18:16:12.302688+0000","last_fullsized":"2024-04-23T18:16:12.302688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301901+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:16:12.301901+0000","last_peered":"2024-04-23T18:16:12.301901+0000","last_clean":"2024-04-23T18:16:12.301901+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:16:12.301901+0000","last_undegraded":"2024-04-23T18:16:12.301901+0000","last_fullsized":"2024-04-23T18:16:12.301901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301607+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:16:12.301607+0000","last_peered":"2024-04-23T18:16:12.301607+0000","last_clean":"2024-04-23T18:16:12.301607+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:16:12.301607+0000","last_undegraded":"2024-04-23T18:16:12.301607+0000","last_fullsized":"2024-04-23T18:16:12.301607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300722+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:16:12.300722+0000","last_peered":"2024-04-23T18:16:12.300722+0000","last_clean":"2024-04-23T18:16:12.300722+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:16:12.300722+0000","last_undegraded":"2024-04-23T18:16:12.300722+0000","last_fullsized":"2024-04-23T18:16:12.300722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300808+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:16:12.300808+0000","last_peered":"2024-04-23T18:16:12.300808+0000","last_clean":"2024-04-23T18:16:12.300808+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:16:12.300808+0000","last_undegraded":"2024-04-23T18:16:12.300808+0000","last_fullsized":"2024-04-23T18:16:12.300808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301627+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:16:12.301627+0000","last_peered":"2024-04-23T18:16:12.301627+0000","last_clean":"2024-04-23T18:16:12.301627+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:16:12.301627+0000","last_undegraded":"2024-04-23T18:16:12.301627+0000","last_fullsized":"2024-04-23T18:16:12.301627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300613+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:16:12.300613+0000","last_peered":"2024-04-23T18:16:12.300613+0000","last_clean":"2024-04-23T18:16:12.300613+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:16:12.300613+0000","last_undegraded":"2024-04-23T18:16:12.300613+0000","last_fullsized":"2024-04-23T18:16:12.300613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65047,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":445,"num_read_kb":289,"num_write":156,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":656,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":460733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1113617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":460935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41028,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704124,"statfs":{"total":95995035648,"available":95953022976,"internally_reserved":0,"allocated":1564672,"data_stored":1052785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":460324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":521163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40456,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704696,"statfs":{"total":95995035648,"available":95953608704,"internally_reserved":0,"allocated":978944,"data_stored":461595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":521857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":460635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40452,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704700,"statfs":{"total":95995035648,"available":95953612800,"internally_reserved":0,"allocated":970752,"data_stored":461774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1576960,"data_stored":1052078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:16:25.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:16:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:25 smithi131 ceph-mon[33041]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:26.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:25 smithi131 ceph-mon[33041]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:26.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:25 smithi026 ceph-mon[29364]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:26.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:25 smithi026 ceph-mon[29364]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:25 smithi110 ceph-mon[32585]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:25 smithi110 ceph-mon[32585]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:27.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 -- 172.21.15.26:0/752565244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434102a90 msgr2=0x7ff434102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 --2- 172.21.15.26:0/752565244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434102a90 0x7ff434102f10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff428009990 tx=0x7ff428009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 -- 172.21.15.26:0/752565244 shutdown_connections 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 --2- 172.21.15.26:0/752565244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 0x7ff43410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 --2- 172.21.15.26:0/752565244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434108a50 0x7ff434108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 --2- 172.21.15.26:0/752565244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434102a90 0x7ff434102f10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 -- 172.21.15.26:0/752565244 >> 172.21.15.26:0/752565244 conn(0x7ff434079ba0 msgr2=0x7ff434079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 -- 172.21.15.26:0/752565244 shutdown_connections 2024-04-23T18:16:27.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.669+0000 7ff4393d2700 1 -- 172.21.15.26:0/752565244 wait complete. 2024-04-23T18:16:27.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 Processor -- start 2024-04-23T18:16:27.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 -- start start 2024-04-23T18:16:27.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 0x7ff434199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 0x7ff43419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112c00 con 0x7ff434108a50 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112a80 con 0x7ff434102a90 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4393d2700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112d80 con 0x7ff434103450 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff432ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4337fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 0x7ff43419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff432ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:44830/0 (socket says 172.21.15.26:44830) 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff432ffd700 1 -- 172.21.15.26:0/186679232 learned_addr learned my addr 172.21.15.26:0/186679232 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.670+0000 7ff4327fc700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 0x7ff434199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 -- 172.21.15.26:0/186679232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 msgr2=0x7ff434199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 0x7ff434199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 -- 172.21.15.26:0/186679232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 msgr2=0x7ff434199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 -- 172.21.15.26:0/186679232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43419e140 con 0x7ff434108a50 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff432ffd700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4337fe700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 0x7ff43419da00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff42c007f10 tx=0x7ff42c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c004db0 con 0x7ff434108a50 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c009e20 con 0x7ff434108a50 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43419e430 con 0x7ff434108a50 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c015400 con 0x7ff434108a50 2024-04-23T18:16:27.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.671+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a5cb0 con 0x7ff434108a50 2024-04-23T18:16:27.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.672+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff434069730 con 0x7ff434108a50 2024-04-23T18:16:27.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.673+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff42c005260 con 0x7ff434108a50 2024-04-23T18:16:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.674+0000 7ff41bfff700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 0x7ff41c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.674+0000 7ff432ffd700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 0x7ff41c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.674+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff42c0cfe00 con 0x7ff434108a50 2024-04-23T18:16:27.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.675+0000 7ff432ffd700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 0x7ff41c0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff424005950 tx=0x7ff42400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:27.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.679+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff42c010070 con 0x7ff434108a50 2024-04-23T18:16:27.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.881+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff434062e80 con 0x7ff41c0a3d50 2024-04-23T18:16:27.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.888+0000 7ff41bfff700 1 -- 172.21.15.26:0/186679232 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258710 (secure 0 0 0) 0x7ff434062e80 con 0x7ff41c0a3d50 2024-04-23T18:16:27.885 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 msgr2=0x7ff41c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 0x7ff41c0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff424005950 tx=0x7ff42400b500 comp rx=0 tx=0).stop 2024-04-23T18:16:27.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 msgr2=0x7ff43419da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 0x7ff43419da00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff42c007f10 tx=0x7ff42c00d3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 shutdown_connections 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff41c0a3d50 0x7ff41c0a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff434108a50 0x7ff43419da00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff434102a90 0x7ff434199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 --2- 172.21.15.26:0/186679232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff434103450 0x7ff434199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 >> 172.21.15.26:0/186679232 conn(0x7ff434079ba0 msgr2=0x7ff4341017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:27.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 shutdown_connections 2024-04-23T18:16:27.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:27.891+0000 7ff4393d2700 1 -- 172.21.15.26:0/186679232 wait complete. 2024-04-23T18:16:27.890 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:16:28.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:27 smithi131 ceph-mon[33041]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:28.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:27 smithi026 ceph-mon[29364]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:28.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:27 smithi110 ceph-mon[32585]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T18:16:28.680 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-23T18:16:26.858422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655649,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":596,"num_read_kb":418,"num_write":339,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181392,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399972,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124541852,"statfs":{"total":1151940427776,"available":1151530856448,"internally_reserved":0,"allocated":13627392,"data_stored":7488351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":38,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301386+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:16:12.301386+0000","last_peered":"2024-04-23T18:16:12.301386+0000","last_clean":"2024-04-23T18:16:12.301386+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:16:12.301386+0000","last_undegraded":"2024-04-23T18:16:12.301386+0000","last_fullsized":"2024-04-23T18:16:12.301386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302779+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:16:12.302779+0000","last_peered":"2024-04-23T18:16:12.302779+0000","last_clean":"2024-04-23T18:16:12.302779+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:16:12.302779+0000","last_undegraded":"2024-04-23T18:16:12.302779+0000","last_fullsized":"2024-04-23T18:16:12.302779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302542+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:16:12.302542+0000","last_peered":"2024-04-23T18:16:12.302542+0000","last_clean":"2024-04-23T18:16:12.302542+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:16:12.302542+0000","last_undegraded":"2024-04-23T18:16:12.302542+0000","last_fullsized":"2024-04-23T18:16:12.302542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301752+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:16:12.301752+0000","last_peered":"2024-04-23T18:16:12.301752+0000","last_clean":"2024-04-23T18:16:12.301752+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:16:12.301752+0000","last_undegraded":"2024-04-23T18:16:12.301752+0000","last_fullsized":"2024-04-23T18:16:12.301752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300906+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:16:12.300906+0000","last_peered":"2024-04-23T18:16:12.300906+0000","last_clean":"2024-04-23T18:16:12.300906+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:16:12.300906+0000","last_undegraded":"2024-04-23T18:16:12.300906+0000","last_fullsized":"2024-04-23T18:16:12.300906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.298749+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:16:12.298749+0000","last_peered":"2024-04-23T18:16:12.298749+0000","last_clean":"2024-04-23T18:16:12.298749+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:16:12.298749+0000","last_undegraded":"2024-04-23T18:16:12.298749+0000","last_fullsized":"2024-04-23T18:16:12.298749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301536+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:16:12.301536+0000","last_peered":"2024-04-23T18:16:12.301536+0000","last_clean":"2024-04-23T18:16:12.301536+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:16:12.301536+0000","last_undegraded":"2024-04-23T18:16:12.301536+0000","last_fullsized":"2024-04-23T18:16:12.301536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300478+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:16:12.300478+0000","last_peered":"2024-04-23T18:16:12.300478+0000","last_clean":"2024-04-23T18:16:12.300478+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:16:12.300478+0000","last_undegraded":"2024-04-23T18:16:12.300478+0000","last_fullsized":"2024-04-23T18:16:12.300478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301082+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:16:12.301082+0000","last_peered":"2024-04-23T18:16:12.301082+0000","last_clean":"2024-04-23T18:16:12.301082+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:16:12.301082+0000","last_undegraded":"2024-04-23T18:16:12.301082+0000","last_fullsized":"2024-04-23T18:16:12.301082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.299182+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:16:12.299182+0000","last_peered":"2024-04-23T18:16:12.299182+0000","last_clean":"2024-04-23T18:16:12.299182+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:16:12.299182+0000","last_undegraded":"2024-04-23T18:16:12.299182+0000","last_fullsized":"2024-04-23T18:16:12.299182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301650+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:16:12.301650+0000","last_peered":"2024-04-23T18:16:12.301650+0000","last_clean":"2024-04-23T18:16:12.301650+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:16:12.301650+0000","last_undegraded":"2024-04-23T18:16:12.301650+0000","last_fullsized":"2024-04-23T18:16:12.301650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301670+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:16:12.301670+0000","last_peered":"2024-04-23T18:16:12.301670+0000","last_clean":"2024-04-23T18:16:12.301670+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:16:12.301670+0000","last_undegraded":"2024-04-23T18:16:12.301670+0000","last_fullsized":"2024-04-23T18:16:12.301670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301538+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:16:12.301538+0000","last_peered":"2024-04-23T18:16:12.301538+0000","last_clean":"2024-04-23T18:16:12.301538+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:16:12.301538+0000","last_undegraded":"2024-04-23T18:16:12.301538+0000","last_fullsized":"2024-04-23T18:16:12.301538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301811+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:16:12.301811+0000","last_peered":"2024-04-23T18:16:12.301811+0000","last_clean":"2024-04-23T18:16:12.301811+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:16:12.301811+0000","last_undegraded":"2024-04-23T18:16:12.301811+0000","last_fullsized":"2024-04-23T18:16:12.301811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302097+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:16:12.302097+0000","last_peered":"2024-04-23T18:16:12.302097+0000","last_clean":"2024-04-23T18:16:12.302097+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:16:12.302097+0000","last_undegraded":"2024-04-23T18:16:12.302097+0000","last_fullsized":"2024-04-23T18:16:12.302097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300531+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:16:12.300531+0000","last_peered":"2024-04-23T18:16:12.300531+0000","last_clean":"2024-04-23T18:16:12.300531+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:16:12.300531+0000","last_undegraded":"2024-04-23T18:16:12.300531+0000","last_fullsized":"2024-04-23T18:16:12.300531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301572+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:16:12.301572+0000","last_peered":"2024-04-23T18:16:12.301572+0000","last_clean":"2024-04-23T18:16:12.301572+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:16:12.301572+0000","last_undegraded":"2024-04-23T18:16:12.301572+0000","last_fullsized":"2024-04-23T18:16:12.301572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297724+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:16:12.297724+0000","last_peered":"2024-04-23T18:16:12.297724+0000","last_clean":"2024-04-23T18:16:12.297724+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:16:12.297724+0000","last_undegraded":"2024-04-23T18:16:12.297724+0000","last_fullsized":"2024-04-23T18:16:12.297724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302289+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:16:12.302289+0000","last_peered":"2024-04-23T18:16:12.302289+0000","last_clean":"2024-04-23T18:16:12.302289+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:16:12.302289+0000","last_undegraded":"2024-04-23T18:16:12.302289+0000","last_fullsized":"2024-04-23T18:16:12.302289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297643+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:16:12.297643+0000","last_peered":"2024-04-23T18:16:12.297643+0000","last_clean":"2024-04-23T18:16:12.297643+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:16:12.297643+0000","last_undegraded":"2024-04-23T18:16:12.297643+0000","last_fullsized":"2024-04-23T18:16:12.297643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301485+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:16:12.301485+0000","last_peered":"2024-04-23T18:16:12.301485+0000","last_clean":"2024-04-23T18:16:12.301485+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:16:12.301485+0000","last_undegraded":"2024-04-23T18:16:12.301485+0000","last_fullsized":"2024-04-23T18:16:12.301485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301775+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:16:12.301775+0000","last_peered":"2024-04-23T18:16:12.301775+0000","last_clean":"2024-04-23T18:16:12.301775+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:16:12.301775+0000","last_undegraded":"2024-04-23T18:16:12.301775+0000","last_fullsized":"2024-04-23T18:16:12.301775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303016+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:16:12.303016+0000","last_peered":"2024-04-23T18:16:12.303016+0000","last_clean":"2024-04-23T18:16:12.303016+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:16:12.303016+0000","last_undegraded":"2024-04-23T18:16:12.303016+0000","last_fullsized":"2024-04-23T18:16:12.303016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304495+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:16:12.304495+0000","last_peered":"2024-04-23T18:16:12.304495+0000","last_clean":"2024-04-23T18:16:12.304495+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:16:12.304495+0000","last_undegraded":"2024-04-23T18:16:12.304495+0000","last_fullsized":"2024-04-23T18:16:12.304495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300889+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:16:12.300889+0000","last_peered":"2024-04-23T18:16:12.300889+0000","last_clean":"2024-04-23T18:16:12.300889+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:16:12.300889+0000","last_undegraded":"2024-04-23T18:16:12.300889+0000","last_fullsized":"2024-04-23T18:16:12.300889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301281+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:16:12.301281+0000","last_peered":"2024-04-23T18:16:12.301281+0000","last_clean":"2024-04-23T18:16:12.301281+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:16:12.301281+0000","last_undegraded":"2024-04-23T18:16:12.301281+0000","last_fullsized":"2024-04-23T18:16:12.301281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302137+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:16:12.302137+0000","last_peered":"2024-04-23T18:16:12.302137+0000","last_clean":"2024-04-23T18:16:12.302137+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:16:12.302137+0000","last_undegraded":"2024-04-23T18:16:12.302137+0000","last_fullsized":"2024-04-23T18:16:12.302137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:21.318849+0000","last_change":"2024-04-23T18:16:12.309950+0000","last_active":"2024-04-23T18:16:21.318849+0000","last_peered":"2024-04-23T18:16:21.318849+0000","last_clean":"2024-04-23T18:16:21.318849+0000","last_became_active":"2024-04-23T18:16:12.305208+0000","last_became_peered":"2024-04-23T18:16:12.305208+0000","last_unstale":"2024-04-23T18:16:21.318849+0000","last_undegraded":"2024-04-23T18:16:21.318849+0000","last_fullsized":"2024-04-23T18:16:21.318849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:02.996009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300811+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:16:12.300811+0000","last_peered":"2024-04-23T18:16:12.300811+0000","last_clean":"2024-04-23T18:16:12.300811+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:16:12.300811+0000","last_undegraded":"2024-04-23T18:16:12.300811+0000","last_fullsized":"2024-04-23T18:16:12.300811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.298721+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:16:12.298721+0000","last_peered":"2024-04-23T18:16:12.298721+0000","last_clean":"2024-04-23T18:16:12.298721+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:16:12.298721+0000","last_undegraded":"2024-04-23T18:16:12.298721+0000","last_fullsized":"2024-04-23T18:16:12.298721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303154+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:16:12.303154+0000","last_peered":"2024-04-23T18:16:12.303154+0000","last_clean":"2024-04-23T18:16:12.303154+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:16:12.303154+0000","last_undegraded":"2024-04-23T18:16:12.303154+0000","last_fullsized":"2024-04-23T18:16:12.303154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300600+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:16:12.300600+0000","last_peered":"2024-04-23T18:16:12.300600+0000","last_clean":"2024-04-23T18:16:12.300600+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:16:12.300600+0000","last_undegraded":"2024-04-23T18:16:12.300600+0000","last_fullsized":"2024-04-23T18:16:12.300600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300513+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:16:12.300513+0000","last_peered":"2024-04-23T18:16:12.300513+0000","last_clean":"2024-04-23T18:16:12.300513+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:16:12.300513+0000","last_undegraded":"2024-04-23T18:16:12.300513+0000","last_fullsized":"2024-04-23T18:16:12.300513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301490+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:16:12.301490+0000","last_peered":"2024-04-23T18:16:12.301490+0000","last_clean":"2024-04-23T18:16:12.301490+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:16:12.301490+0000","last_undegraded":"2024-04-23T18:16:12.301490+0000","last_fullsized":"2024-04-23T18:16:12.301490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297531+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:16:12.297531+0000","last_peered":"2024-04-23T18:16:12.297531+0000","last_clean":"2024-04-23T18:16:12.297531+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:16:12.297531+0000","last_undegraded":"2024-04-23T18:16:12.297531+0000","last_fullsized":"2024-04-23T18:16:12.297531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:24.322468+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:16:24.322468+0000","last_peered":"2024-04-23T18:16:24.322468+0000","last_clean":"2024-04-23T18:16:24.322468+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:16:24.322468+0000","last_undegraded":"2024-04-23T18:16:24.322468+0000","last_fullsized":"2024-04-23T18:16:24.322468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302279+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:16:12.302279+0000","last_peered":"2024-04-23T18:16:12.302279+0000","last_clean":"2024-04-23T18:16:12.302279+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:16:12.302279+0000","last_undegraded":"2024-04-23T18:16:12.302279+0000","last_fullsized":"2024-04-23T18:16:12.302279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301850+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:16:12.301850+0000","last_peered":"2024-04-23T18:16:12.301850+0000","last_clean":"2024-04-23T18:16:12.301850+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:16:12.301850+0000","last_undegraded":"2024-04-23T18:16:12.301850+0000","last_fullsized":"2024-04-23T18:16:12.301850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.306548+0000","last_change":"2024-04-23T18:16:12.306548+0000","last_active":"2024-04-23T18:16:12.306548+0000","last_peered":"2024-04-23T18:16:12.306548+0000","last_clean":"2024-04-23T18:16:12.306548+0000","last_became_active":"2024-04-23T18:16:12.306227+0000","last_became_peered":"2024-04-23T18:16:12.306227+0000","last_unstale":"2024-04-23T18:16:12.306548+0000","last_undegraded":"2024-04-23T18:16:12.306548+0000","last_fullsized":"2024-04-23T18:16:12.306548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:49.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302291+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:16:12.302291+0000","last_peered":"2024-04-23T18:16:12.302291+0000","last_clean":"2024-04-23T18:16:12.302291+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:16:12.302291+0000","last_undegraded":"2024-04-23T18:16:12.302291+0000","last_fullsized":"2024-04-23T18:16:12.302291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304182+0000","last_change":"2024-04-23T18:16:12.304182+0000","last_active":"2024-04-23T18:16:12.304182+0000","last_peered":"2024-04-23T18:16:12.304182+0000","last_clean":"2024-04-23T18:16:12.304182+0000","last_became_active":"2024-04-23T18:16:12.303708+0000","last_became_peered":"2024-04-23T18:16:12.303708+0000","last_unstale":"2024-04-23T18:16:12.304182+0000","last_undegraded":"2024-04-23T18:16:12.304182+0000","last_fullsized":"2024-04-23T18:16:12.304182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:32.224284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'28","reported_seq":203,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:25.323957+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:16:25.323957+0000","last_peered":"2024-04-23T18:16:25.323957+0000","last_clean":"2024-04-23T18:16:25.323957+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:16:25.323957+0000","last_undegraded":"2024-04-23T18:16:25.323957+0000","last_fullsized":"2024-04-23T18:16:25.323957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":156,"num_read_kb":137,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304348+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:16:12.304348+0000","last_peered":"2024-04-23T18:16:12.304348+0000","last_clean":"2024-04-23T18:16:12.304348+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:16:12.304348+0000","last_undegraded":"2024-04-23T18:16:12.304348+0000","last_fullsized":"2024-04-23T18:16:12.304348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.314870+0000","last_change":"2024-04-23T18:16:12.314402+0000","last_active":"2024-04-23T18:16:12.314870+0000","last_peered":"2024-04-23T18:16:12.314870+0000","last_clean":"2024-04-23T18:16:12.314870+0000","last_became_active":"2024-04-23T18:16:12.305588+0000","last_became_peered":"2024-04-23T18:16:12.305588+0000","last_unstale":"2024-04-23T18:16:12.314870+0000","last_undegraded":"2024-04-23T18:16:12.314870+0000","last_fullsized":"2024-04-23T18:16:12.314870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:15.413380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301245+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:16:12.301245+0000","last_peered":"2024-04-23T18:16:12.301245+0000","last_clean":"2024-04-23T18:16:12.301245+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:16:12.301245+0000","last_undegraded":"2024-04-23T18:16:12.301245+0000","last_fullsized":"2024-04-23T18:16:12.301245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301274+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:16:12.301274+0000","last_peered":"2024-04-23T18:16:12.301274+0000","last_clean":"2024-04-23T18:16:12.301274+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:16:12.301274+0000","last_undegraded":"2024-04-23T18:16:12.301274+0000","last_fullsized":"2024-04-23T18:16:12.301274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302760+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:16:12.302760+0000","last_peered":"2024-04-23T18:16:12.302760+0000","last_clean":"2024-04-23T18:16:12.302760+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:16:12.302760+0000","last_undegraded":"2024-04-23T18:16:12.302760+0000","last_fullsized":"2024-04-23T18:16:12.302760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":99,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:22.776995+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:16:22.776995+0000","last_peered":"2024-04-23T18:16:22.776995+0000","last_clean":"2024-04-23T18:16:22.776995+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:16:22.776995+0000","last_undegraded":"2024-04-23T18:16:22.776995+0000","last_fullsized":"2024-04-23T18:16:22.776995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301219+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:16:12.301219+0000","last_peered":"2024-04-23T18:16:12.301219+0000","last_clean":"2024-04-23T18:16:12.301219+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:16:12.301219+0000","last_undegraded":"2024-04-23T18:16:12.301219+0000","last_fullsized":"2024-04-23T18:16:12.301219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300810+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:16:12.300810+0000","last_peered":"2024-04-23T18:16:12.300810+0000","last_clean":"2024-04-23T18:16:12.300810+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:16:12.300810+0000","last_undegraded":"2024-04-23T18:16:12.300810+0000","last_fullsized":"2024-04-23T18:16:12.300810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300937+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:16:12.300937+0000","last_peered":"2024-04-23T18:16:12.300937+0000","last_clean":"2024-04-23T18:16:12.300937+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:16:12.300937+0000","last_undegraded":"2024-04-23T18:16:12.300937+0000","last_fullsized":"2024-04-23T18:16:12.300937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304603+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:16:12.304603+0000","last_peered":"2024-04-23T18:16:12.304603+0000","last_clean":"2024-04-23T18:16:12.304603+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:16:12.304603+0000","last_undegraded":"2024-04-23T18:16:12.304603+0000","last_fullsized":"2024-04-23T18:16:12.304603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304948+0000","last_change":"2024-04-23T18:16:12.304948+0000","last_active":"2024-04-23T18:16:12.304948+0000","last_peered":"2024-04-23T18:16:12.304948+0000","last_clean":"2024-04-23T18:16:12.304948+0000","last_became_active":"2024-04-23T18:16:12.304541+0000","last_became_peered":"2024-04-23T18:16:12.304541+0000","last_unstale":"2024-04-23T18:16:12.304948+0000","last_undegraded":"2024-04-23T18:16:12.304948+0000","last_fullsized":"2024-04-23T18:16:12.304948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:47.148967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297738+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:16:12.297738+0000","last_peered":"2024-04-23T18:16:12.297738+0000","last_clean":"2024-04-23T18:16:12.297738+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:16:12.297738+0000","last_undegraded":"2024-04-23T18:16:12.297738+0000","last_fullsized":"2024-04-23T18:16:12.297738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.299155+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:16:12.299155+0000","last_peered":"2024-04-23T18:16:12.299155+0000","last_clean":"2024-04-23T18:16:12.299155+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:16:12.299155+0000","last_undegraded":"2024-04-23T18:16:12.299155+0000","last_fullsized":"2024-04-23T18:16:12.299155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301910+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:16:12.301910+0000","last_peered":"2024-04-23T18:16:12.301910+0000","last_clean":"2024-04-23T18:16:12.301910+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:16:12.301910+0000","last_undegraded":"2024-04-23T18:16:12.301910+0000","last_fullsized":"2024-04-23T18:16:12.301910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304513+0000","last_change":"2024-04-23T18:16:12.304513+0000","last_active":"2024-04-23T18:16:12.304513+0000","last_peered":"2024-04-23T18:16:12.304513+0000","last_clean":"2024-04-23T18:16:12.304513+0000","last_became_active":"2024-04-23T18:16:12.304069+0000","last_became_peered":"2024-04-23T18:16:12.304069+0000","last_unstale":"2024-04-23T18:16:12.304513+0000","last_undegraded":"2024-04-23T18:16:12.304513+0000","last_fullsized":"2024-04-23T18:16:12.304513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:58.834388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301712+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:16:12.301712+0000","last_peered":"2024-04-23T18:16:12.301712+0000","last_clean":"2024-04-23T18:16:12.301712+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:16:12.301712+0000","last_undegraded":"2024-04-23T18:16:12.301712+0000","last_fullsized":"2024-04-23T18:16:12.301712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:20.772445+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:16:20.772445+0000","last_peered":"2024-04-23T18:16:20.772445+0000","last_clean":"2024-04-23T18:16:20.772445+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:16:20.772445+0000","last_undegraded":"2024-04-23T18:16:20.772445+0000","last_fullsized":"2024-04-23T18:16:20.772445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301942+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:16:12.301942+0000","last_peered":"2024-04-23T18:16:12.301942+0000","last_clean":"2024-04-23T18:16:12.301942+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:16:12.301942+0000","last_undegraded":"2024-04-23T18:16:12.301942+0000","last_fullsized":"2024-04-23T18:16:12.301942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301317+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:16:12.301317+0000","last_peered":"2024-04-23T18:16:12.301317+0000","last_clean":"2024-04-23T18:16:12.301317+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:16:12.301317+0000","last_undegraded":"2024-04-23T18:16:12.301317+0000","last_fullsized":"2024-04-23T18:16:12.301317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301459+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:16:12.301459+0000","last_peered":"2024-04-23T18:16:12.301459+0000","last_clean":"2024-04-23T18:16:12.301459+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:16:12.301459+0000","last_undegraded":"2024-04-23T18:16:12.301459+0000","last_fullsized":"2024-04-23T18:16:12.301459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300375+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:16:12.300375+0000","last_peered":"2024-04-23T18:16:12.300375+0000","last_clean":"2024-04-23T18:16:12.300375+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:16:12.300375+0000","last_undegraded":"2024-04-23T18:16:12.300375+0000","last_fullsized":"2024-04-23T18:16:12.300375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301684+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:16:12.301684+0000","last_peered":"2024-04-23T18:16:12.301684+0000","last_clean":"2024-04-23T18:16:12.301684+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:16:12.301684+0000","last_undegraded":"2024-04-23T18:16:12.301684+0000","last_fullsized":"2024-04-23T18:16:12.301684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304385+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:16:12.304385+0000","last_peered":"2024-04-23T18:16:12.304385+0000","last_clean":"2024-04-23T18:16:12.304385+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:16:12.304385+0000","last_undegraded":"2024-04-23T18:16:12.304385+0000","last_fullsized":"2024-04-23T18:16:12.304385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301325+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:16:12.301325+0000","last_peered":"2024-04-23T18:16:12.301325+0000","last_clean":"2024-04-23T18:16:12.301325+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:16:12.301325+0000","last_undegraded":"2024-04-23T18:16:12.301325+0000","last_fullsized":"2024-04-23T18:16:12.301325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.297756+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:16:12.297756+0000","last_peered":"2024-04-23T18:16:12.297756+0000","last_clean":"2024-04-23T18:16:12.297756+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:16:12.297756+0000","last_undegraded":"2024-04-23T18:16:12.297756+0000","last_fullsized":"2024-04-23T18:16:12.297756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301997+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:16:12.301997+0000","last_peered":"2024-04-23T18:16:12.301997+0000","last_clean":"2024-04-23T18:16:12.301997+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:16:12.301997+0000","last_undegraded":"2024-04-23T18:16:12.301997+0000","last_fullsized":"2024-04-23T18:16:12.301997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301113+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:16:12.301113+0000","last_peered":"2024-04-23T18:16:12.301113+0000","last_clean":"2024-04-23T18:16:12.301113+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:16:12.301113+0000","last_undegraded":"2024-04-23T18:16:12.301113+0000","last_fullsized":"2024-04-23T18:16:12.301113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302832+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:16:12.302832+0000","last_peered":"2024-04-23T18:16:12.302832+0000","last_clean":"2024-04-23T18:16:12.302832+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:16:12.302832+0000","last_undegraded":"2024-04-23T18:16:12.302832+0000","last_fullsized":"2024-04-23T18:16:12.302832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300544+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:16:12.300544+0000","last_peered":"2024-04-23T18:16:12.300544+0000","last_clean":"2024-04-23T18:16:12.300544+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:16:12.300544+0000","last_undegraded":"2024-04-23T18:16:12.300544+0000","last_fullsized":"2024-04-23T18:16:12.300544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300961+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:16:12.300961+0000","last_peered":"2024-04-23T18:16:12.300961+0000","last_clean":"2024-04-23T18:16:12.300961+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:16:12.300961+0000","last_undegraded":"2024-04-23T18:16:12.300961+0000","last_fullsized":"2024-04-23T18:16:12.300961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300731+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:16:12.300731+0000","last_peered":"2024-04-23T18:16:12.300731+0000","last_clean":"2024-04-23T18:16:12.300731+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:16:12.300731+0000","last_undegraded":"2024-04-23T18:16:12.300731+0000","last_fullsized":"2024-04-23T18:16:12.300731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301401+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:16:12.301401+0000","last_peered":"2024-04-23T18:16:12.301401+0000","last_clean":"2024-04-23T18:16:12.301401+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:16:12.301401+0000","last_undegraded":"2024-04-23T18:16:12.301401+0000","last_fullsized":"2024-04-23T18:16:12.301401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301615+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:16:12.301615+0000","last_peered":"2024-04-23T18:16:12.301615+0000","last_clean":"2024-04-23T18:16:12.301615+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:16:12.301615+0000","last_undegraded":"2024-04-23T18:16:12.301615+0000","last_fullsized":"2024-04-23T18:16:12.301615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.304844+0000","last_change":"2024-04-23T18:16:12.304138+0000","last_active":"2024-04-23T18:16:12.304844+0000","last_peered":"2024-04-23T18:16:12.304844+0000","last_clean":"2024-04-23T18:16:12.304844+0000","last_became_active":"2024-04-23T18:16:12.303760+0000","last_became_peered":"2024-04-23T18:16:12.303760+0000","last_unstale":"2024-04-23T18:16:12.304844+0000","last_undegraded":"2024-04-23T18:16:12.304844+0000","last_fullsized":"2024-04-23T18:16:12.304844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:10.049511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.318277+0000","last_change":"2024-04-23T18:16:12.317810+0000","last_active":"2024-04-23T18:16:12.318277+0000","last_peered":"2024-04-23T18:16:12.318277+0000","last_clean":"2024-04-23T18:16:12.318277+0000","last_became_active":"2024-04-23T18:16:12.307064+0000","last_became_peered":"2024-04-23T18:16:12.307064+0000","last_unstale":"2024-04-23T18:16:12.318277+0000","last_undegraded":"2024-04-23T18:16:12.318277+0000","last_fullsized":"2024-04-23T18:16:12.318277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:06:41.038382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302024+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:16:12.302024+0000","last_peered":"2024-04-23T18:16:12.302024+0000","last_clean":"2024-04-23T18:16:12.302024+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:16:12.302024+0000","last_undegraded":"2024-04-23T18:16:12.302024+0000","last_fullsized":"2024-04-23T18:16:12.302024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301974+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:16:12.301974+0000","last_peered":"2024-04-23T18:16:12.301974+0000","last_clean":"2024-04-23T18:16:12.301974+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:16:12.301974+0000","last_undegraded":"2024-04-23T18:16:12.301974+0000","last_fullsized":"2024-04-23T18:16:12.301974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301713+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:16:12.301713+0000","last_peered":"2024-04-23T18:16:12.301713+0000","last_clean":"2024-04-23T18:16:12.301713+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:16:12.301713+0000","last_undegraded":"2024-04-23T18:16:12.301713+0000","last_fullsized":"2024-04-23T18:16:12.301713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300940+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:16:12.300940+0000","last_peered":"2024-04-23T18:16:12.300940+0000","last_clean":"2024-04-23T18:16:12.300940+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:16:12.300940+0000","last_undegraded":"2024-04-23T18:16:12.300940+0000","last_fullsized":"2024-04-23T18:16:12.300940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301454+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:16:12.301454+0000","last_peered":"2024-04-23T18:16:12.301454+0000","last_clean":"2024-04-23T18:16:12.301454+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:16:12.301454+0000","last_undegraded":"2024-04-23T18:16:12.301454+0000","last_fullsized":"2024-04-23T18:16:12.301454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301135+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:16:12.301135+0000","last_peered":"2024-04-23T18:16:12.301135+0000","last_clean":"2024-04-23T18:16:12.301135+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:16:12.301135+0000","last_undegraded":"2024-04-23T18:16:12.301135+0000","last_fullsized":"2024-04-23T18:16:12.301135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302422+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:16:12.302422+0000","last_peered":"2024-04-23T18:16:12.302422+0000","last_clean":"2024-04-23T18:16:12.302422+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:16:12.302422+0000","last_undegraded":"2024-04-23T18:16:12.302422+0000","last_fullsized":"2024-04-23T18:16:12.302422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301119+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:16:12.301119+0000","last_peered":"2024-04-23T18:16:12.301119+0000","last_clean":"2024-04-23T18:16:12.301119+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:16:12.301119+0000","last_undegraded":"2024-04-23T18:16:12.301119+0000","last_fullsized":"2024-04-23T18:16:12.301119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301385+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:16:12.301385+0000","last_peered":"2024-04-23T18:16:12.301385+0000","last_clean":"2024-04-23T18:16:12.301385+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:16:12.301385+0000","last_undegraded":"2024-04-23T18:16:12.301385+0000","last_fullsized":"2024-04-23T18:16:12.301385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301516+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:16:12.301516+0000","last_peered":"2024-04-23T18:16:12.301516+0000","last_clean":"2024-04-23T18:16:12.301516+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:16:12.301516+0000","last_undegraded":"2024-04-23T18:16:12.301516+0000","last_fullsized":"2024-04-23T18:16:12.301516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.303277+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:16:12.303277+0000","last_peered":"2024-04-23T18:16:12.303277+0000","last_clean":"2024-04-23T18:16:12.303277+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:16:12.303277+0000","last_undegraded":"2024-04-23T18:16:12.303277+0000","last_fullsized":"2024-04-23T18:16:12.303277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301421+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:16:12.301421+0000","last_peered":"2024-04-23T18:16:12.301421+0000","last_clean":"2024-04-23T18:16:12.301421+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:16:12.301421+0000","last_undegraded":"2024-04-23T18:16:12.301421+0000","last_fullsized":"2024-04-23T18:16:12.301421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301491+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:16:12.301491+0000","last_peered":"2024-04-23T18:16:12.301491+0000","last_clean":"2024-04-23T18:16:12.301491+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:16:12.301491+0000","last_undegraded":"2024-04-23T18:16:12.301491+0000","last_fullsized":"2024-04-23T18:16:12.301491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.302688+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:16:12.302688+0000","last_peered":"2024-04-23T18:16:12.302688+0000","last_clean":"2024-04-23T18:16:12.302688+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:16:12.302688+0000","last_undegraded":"2024-04-23T18:16:12.302688+0000","last_fullsized":"2024-04-23T18:16:12.302688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301901+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:16:12.301901+0000","last_peered":"2024-04-23T18:16:12.301901+0000","last_clean":"2024-04-23T18:16:12.301901+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:16:12.301901+0000","last_undegraded":"2024-04-23T18:16:12.301901+0000","last_fullsized":"2024-04-23T18:16:12.301901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301607+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:16:12.301607+0000","last_peered":"2024-04-23T18:16:12.301607+0000","last_clean":"2024-04-23T18:16:12.301607+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:16:12.301607+0000","last_undegraded":"2024-04-23T18:16:12.301607+0000","last_fullsized":"2024-04-23T18:16:12.301607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300722+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:16:12.300722+0000","last_peered":"2024-04-23T18:16:12.300722+0000","last_clean":"2024-04-23T18:16:12.300722+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:16:12.300722+0000","last_undegraded":"2024-04-23T18:16:12.300722+0000","last_fullsized":"2024-04-23T18:16:12.300722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300808+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:16:12.300808+0000","last_peered":"2024-04-23T18:16:12.300808+0000","last_clean":"2024-04-23T18:16:12.300808+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:16:12.300808+0000","last_undegraded":"2024-04-23T18:16:12.300808+0000","last_fullsized":"2024-04-23T18:16:12.300808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.301627+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:16:12.301627+0000","last_peered":"2024-04-23T18:16:12.301627+0000","last_clean":"2024-04-23T18:16:12.301627+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:16:12.301627+0000","last_undegraded":"2024-04-23T18:16:12.301627+0000","last_fullsized":"2024-04-23T18:16:12.301627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:12.300613+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:16:12.300613+0000","last_peered":"2024-04-23T18:16:12.300613+0000","last_clean":"2024-04-23T18:16:12.300613+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:16:12.300613+0000","last_undegraded":"2024-04-23T18:16:12.300613+0000","last_fullsized":"2024-04-23T18:16:12.300613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65047,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":460,"num_read_kb":299,"num_write":156,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":656,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":460733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1113617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":460935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41028,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704124,"statfs":{"total":95995035648,"available":95953022976,"internally_reserved":0,"allocated":1564672,"data_stored":1052785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":460324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":521163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40456,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704696,"statfs":{"total":95995035648,"available":95953608704,"internally_reserved":0,"allocated":978944,"data_stored":461595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":521857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":460635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40452,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704700,"statfs":{"total":95995035648,"available":95953612800,"internally_reserved":0,"allocated":970752,"data_stored":461774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1576960,"data_stored":1052078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:16:28.684 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T18:16:28.684 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T18:16:28.685 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T18:16:28.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T18:16:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:29.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:28 smithi026 ceph-mon[29364]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:28 smithi131 ceph-mon[33041]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:28 smithi110 ceph-mon[32585]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:30.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:29 smithi026 ceph-mon[29364]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:30.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:29 smithi131 ceph-mon[33041]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:30.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:29 smithi110 ceph-mon[32585]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/237746077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 msgr2=0x7f0cb4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/237746077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4101dc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009990 tx=0x7f0cb0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/237746077 shutdown_connections 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/237746077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4102300 0x7f0cb410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/237746077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4107990 0x7f0cb4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/237746077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4101dc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/237746077 >> 172.21.15.26:0/237746077 conn(0x7f0cb40788f0 msgr2=0x7f0cb4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/237746077 shutdown_connections 2024-04-23T18:16:30.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.720+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/237746077 wait complete. 2024-04-23T18:16:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 Processor -- start 2024-04-23T18:16:30.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 -- start start 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 0x7f0cb4199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4107990 0x7f0cb419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb4111bd0 con 0x7f0cb4101940 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb4111a50 con 0x7f0cb4107990 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cbb5d4700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb4111d50 con 0x7f0cb4102300 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cb9370700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cb9370700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38432/0 (socket says 172.21.15.26:38432) 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cb9370700 1 -- 172.21.15.26:0/1290593735 learned_addr learned my addr 172.21.15.26:0/1290593735 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.721+0000 7f0cb9b71700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4107990 0x7f0cb419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb8b6f700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 0x7f0cb4199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 -- 172.21.15.26:0/1290593735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 msgr2=0x7f0cb4199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 0x7f0cb4199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 -- 172.21.15.26:0/1290593735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4107990 msgr2=0x7f0cb419dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4107990 0x7f0cb419dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 -- 172.21.15.26:0/1290593735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb419e310 con 0x7f0cb4101940 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb8b6f700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 0x7f0cb4199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cb9370700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b530 tx=0x7f0ca800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8015020 con 0x7f0cb4101940 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb419e600 con 0x7f0cb4101940 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca8011ce0 con 0x7f0cb4101940 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8010e40 con 0x7f0cb4101940 2024-04-23T18:16:30.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.722+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb41a5e80 con 0x7f0cb4101940 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.723+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb4069730 con 0x7f0cb4101940 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.724+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0ca8018020 con 0x7f0cb4101940 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.725+0000 7f0ca67fc700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 0x7f0ca00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.725+0000 7f0cb8b6f700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 0x7f0ca00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.725+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0ca80cfb20 con 0x7f0cb4101940 2024-04-23T18:16:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.725+0000 7f0cb8b6f700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 0x7f0ca00a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000b5c0 tx=0x7f0cb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.729+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ca801f020 con 0x7f0cb4101940 2024-04-23T18:16:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.985+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0cb419a660 con 0x7f0cb4101940 2024-04-23T18:16:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.986+0000 7f0ca67fc700 1 -- 172.21.15.26:0/1290593735 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0ca809c460 con 0x7f0cb4101940 2024-04-23T18:16:30.979 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:30.980 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 msgr2=0x7f0ca00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 0x7f0ca00a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000b5c0 tx=0x7f0cb0005fb0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 msgr2=0x7f0cb4199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b530 tx=0x7f0ca800b840 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 shutdown_connections 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0ca00a3cd0 0x7f0ca00a6190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cb4107990 0x7f0cb419dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cb4101940 0x7f0cb4199350 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 --2- 172.21.15.26:0/1290593735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cb4102300 0x7f0cb4199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 >> 172.21.15.26:0/1290593735 conn(0x7f0cb40788f0 msgr2=0x7f0cb406aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 shutdown_connections 2024-04-23T18:16:30.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:30.988+0000 7f0cbb5d4700 1 -- 172.21.15.26:0/1290593735 wait complete. 2024-04-23T18:16:31.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:30 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1290593735' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:16:31.306 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:30 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1290593735' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:16:31.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:30 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1290593735' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:16:31.649 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T18:16:31.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- 172.21.15.26:0/1386835758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4107c90 msgr2=0x7f16c410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- 172.21.15.26:0/1386835758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4107c90 0x7f16c410a0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009900 tx=0x7f16b402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- 172.21.15.26:0/1386835758 shutdown_connections 2024-04-23T18:16:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- 172.21.15.26:0/1386835758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4105360 0x7f16c4107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- 172.21.15.26:0/1386835758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4107c90 0x7f16c410a0e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- 172.21.15.26:0/1386835758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4102a30 0x7f16c4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- 172.21.15.26:0/1386835758 >> 172.21.15.26:0/1386835758 conn(0x7f16c40fc440 msgr2=0x7f16c40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- 172.21.15.26:0/1386835758 shutdown_connections 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- 172.21.15.26:0/1386835758 wait complete. 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 Processor -- start 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 -- start start 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4102a30 0x7f16c41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.768+0000 7f16cad19640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4107c90 0x7f16c419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16cad19640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f16c410db50 con 0x7f16c4105360 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16cad19640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f16c410d9d0 con 0x7f16c4102a30 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16cad19640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f16c410dcd0 con 0x7f16c4107c90 2024-04-23T18:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38454/0 (socket says 172.21.15.26:38454) 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 -- 172.21.15.26:0/2526962531 learned_addr learned my addr 172.21.15.26:0/2526962531 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c8ab5640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4102a30 0x7f16c41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c92b6640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4107c90 0x7f16c419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 -- 172.21.15.26:0/2526962531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4107c90 msgr2=0x7f16c419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4107c90 0x7f16c419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 -- 172.21.15.26:0/2526962531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4102a30 msgr2=0x7f16c41981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4102a30 0x7f16c41981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 -- 172.21.15.26:0/2526962531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c419fe10 con 0x7f16c4105360 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c3fff640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16b800d770 tx=0x7f16b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8017070 con 0x7f16c4105360 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b8009c30 con 0x7f16c4105360 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c41a0100 con 0x7f16c4105360 2024-04-23T18:16:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b800f620 con 0x7f16c4105360 2024-04-23T18:16:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.769+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c41a0720 con 0x7f16c4105360 2024-04-23T18:16:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.771+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c40fdf30 con 0x7f16c4105360 2024-04-23T18:16:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.772+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f16b801a070 con 0x7f16c4105360 2024-04-23T18:16:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.772+0000 7f16c1ffb640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 0x7f16ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.773+0000 7f16c8ab5640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 0x7f16ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.773+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f16b80cffc0 con 0x7f16c4105360 2024-04-23T18:16:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.773+0000 7f16c8ab5640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 0x7f16ac0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00a400 tx=0x7f16bc00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.777+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16b8009de0 con 0x7f16c4105360 2024-04-23T18:16:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.970+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16c402d810 con 0x7f16c4105360 2024-04-23T18:16:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.971+0000 7f16c1ffb640 1 -- 172.21.15.26:0/2526962531 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7f16b809c8f0 con 0x7f16c4105360 2024-04-23T18:16:31.965 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:31.965 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 msgr2=0x7f16ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 0x7f16ac0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00a400 tx=0x7f16bc00d040 comp rx=0 tx=0).stop 2024-04-23T18:16:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 msgr2=0x7f16c4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16b800d770 tx=0x7f16b800dc30 comp rx=0 tx=0).stop 2024-04-23T18:16:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 shutdown_connections 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f16ac0a3cc0 0x7f16ac0a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16c4107c90 0x7f16c419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f16c4105360 0x7f16c4198730 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 --2- 172.21.15.26:0/2526962531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c4102a30 0x7f16c41981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.973+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 >> 172.21.15.26:0/2526962531 conn(0x7f16c40fc440 msgr2=0x7f16c4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.974+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 shutdown_connections 2024-04-23T18:16:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:31.974+0000 7f16cad19640 1 -- 172.21.15.26:0/2526962531 wait complete. 2024-04-23T18:16:31.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:16:31.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 -- 172.21.15.26:0/4271743464 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc101fb0 msgr2=0x7f97dc1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 --2- 172.21.15.26:0/4271743464 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc101fb0 0x7f97dc1023d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009900 tx=0x7f97d802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 -- 172.21.15.26:0/4271743464 shutdown_connections 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 --2- 172.21.15.26:0/4271743464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97dc1031a0 0x7f97dc103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 --2- 172.21.15.26:0/4271743464 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc103b60 0x7f97dc10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 --2- 172.21.15.26:0/4271743464 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc101fb0 0x7f97dc1023d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 -- 172.21.15.26:0/4271743464 >> 172.21.15.26:0/4271743464 conn(0x7f97dc0fd520 msgr2=0x7f97dc0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 -- 172.21.15.26:0/4271743464 shutdown_connections 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.091+0000 7f97e4720640 1 -- 172.21.15.26:0/4271743464 wait complete. 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 Processor -- start 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 -- start start 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc101fb0 0x7f97dc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97dc103b60 0x7f97dc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc10daf0 con 0x7f97dc1031a0 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc10d970 con 0x7f97dc103b60 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e4720640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc10dc70 con 0x7f97dc101fb0 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38466/0 (socket says 172.21.15.26:38466) 2024-04-23T18:16:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 -- 172.21.15.26:0/870600061 learned_addr learned my addr 172.21.15.26:0/870600061 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e24bc640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc101fb0 0x7f97dc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 -- 172.21.15.26:0/870600061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc101fb0 msgr2=0x7f97dc198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc101fb0 0x7f97dc198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 -- 172.21.15.26:0/870600061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97dc103b60 msgr2=0x7f97dc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97dc103b60 0x7f97dc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 -- 172.21.15.26:0/870600061 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97dc19fda0 con 0x7f97dc1031a0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97e1cbb640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c920 tx=0x7f97cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.092+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc007c70 con 0x7f97dc1031a0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.093+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97cc007e10 con 0x7f97dc1031a0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.093+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97dc1a0090 con 0x7f97dc1031a0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.093+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc018820 con 0x7f97dc1031a0 2024-04-23T18:16:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.093+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97dc1a05e0 con 0x7f97dc1031a0 2024-04-23T18:16:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.093+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97dc109370 con 0x7f97dc1031a0 2024-04-23T18:16:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.095+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f97cc00f450 con 0x7f97dc1031a0 2024-04-23T18:16:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.096+0000 7f97d37fe640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 0x7f97c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.096+0000 7f97e24bc640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 0x7f97c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.096+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f97cc0d09e0 con 0x7f97dc1031a0 2024-04-23T18:16:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.096+0000 7f97e24bc640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 0x7f97c80a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97d8000c00 tx=0x7f97d80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.099+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97cc014030 con 0x7f97dc1031a0 2024-04-23T18:16:32.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:31 smithi131 ceph-mon[33041]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:32.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:31 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2526962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.287+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97dc02d5c0 con 0x7f97dc1031a0 2024-04-23T18:16:32.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.287+0000 7f97d37fe640 1 -- 172.21.15.26:0/870600061 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f97cc09d300 con 0x7f97dc1031a0 2024-04-23T18:16:32.281 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:16:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 msgr2=0x7f97c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 0x7f97c80a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97d8000c00 tx=0x7f97d80314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 msgr2=0x7f97dc1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c920 tx=0x7f97cc00cde0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 shutdown_connections 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f97c80a3c40 0x7f97c80a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97dc103b60 0x7f97dc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97dc1031a0 0x7f97dc1986c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 --2- 172.21.15.26:0/870600061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97dc101fb0 0x7f97dc198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 >> 172.21.15.26:0/870600061 conn(0x7f97dc0fd520 msgr2=0x7f97dc106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 shutdown_connections 2024-04-23T18:16:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.289+0000 7f97e4720640 1 -- 172.21.15.26:0/870600061 wait complete. 2024-04-23T18:16:32.293 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:31 smithi026 ceph-mon[29364]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:32.293 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:31 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2526962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:16:32.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 -- 172.21.15.26:0/3491518878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4101f80 msgr2=0x7f15e41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 --2- 172.21.15.26:0/3491518878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4101f80 0x7f15e41023a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009900 tx=0x7f15d402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 -- 172.21.15.26:0/3491518878 shutdown_connections 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 --2- 172.21.15.26:0/3491518878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4103170 0x7f15e41035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 --2- 172.21.15.26:0/3491518878 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103b30 0x7f15e410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 --2- 172.21.15.26:0/3491518878 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4101f80 0x7f15e41023a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 -- 172.21.15.26:0/3491518878 >> 172.21.15.26:0/3491518878 conn(0x7f15e40fd510 msgr2=0x7f15e40ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 -- 172.21.15.26:0/3491518878 shutdown_connections 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.406+0000 7f15e8908640 1 -- 172.21.15.26:0/3491518878 wait complete. 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 Processor -- start 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 -- start start 2024-04-23T18:16:32.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4101f80 0x7f15e41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 0x7f15e4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f15e410dac0 con 0x7f15e4103b30 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f15e410d940 con 0x7f15e4101f80 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f15e410dc40 con 0x7f15e4103170 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38496/0 (socket says 172.21.15.26:38496) 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 -- 172.21.15.26:0/440615745 learned_addr learned my addr 172.21.15.26:0/440615745 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e259c640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4101f80 0x7f15e41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e1d9b640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 0x7f15e4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 -- 172.21.15.26:0/440615745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 msgr2=0x7f15e4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 0x7f15e4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 -- 172.21.15.26:0/440615745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4101f80 msgr2=0x7f15e41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4101f80 0x7f15e41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 -- 172.21.15.26:0/440615745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e419fde0 con 0x7f15e4103b30 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e1d9b640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 0x7f15e4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e2d9d640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00cad0 tx=0x7f15dc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00ae20 con 0x7f15e4103b30 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc005020 con 0x7f15e4103b30 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.407+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e41a00d0 con 0x7f15e4103b30 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.408+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00f6b0 con 0x7f15e4103b30 2024-04-23T18:16:32.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.408+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e41a0670 con 0x7f15e4103b30 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.409+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e4109370 con 0x7f15e4103b30 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.410+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f15dc0051d0 con 0x7f15e4103b30 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.410+0000 7f15d37fe640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 0x7f15cc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.410+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f15dc0d0530 con 0x7f15e4103b30 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.410+0000 7f15e259c640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 0x7f15cc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.411+0000 7f15e259c640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 0x7f15cc0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009230 tx=0x7f15d40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.415+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15dc014030 con 0x7f15e4103b30 2024-04-23T18:16:32.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:31 smithi110 ceph-mon[32585]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:32.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:31 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2526962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.603+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15e4062b50 con 0x7f15e4103b30 2024-04-23T18:16:32.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.604+0000 7f15d37fe640 1 -- 172.21.15.26:0/440615745 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f15dc09cdc0 con 0x7f15e4103b30 2024-04-23T18:16:32.597 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 msgr2=0x7f15cc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 0x7f15cc0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009230 tx=0x7f15d40314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 msgr2=0x7f15e419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00cad0 tx=0x7f15dc00cf90 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 shutdown_connections 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f15cc0a3cc0 0x7f15cc0a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f15e4103b30 0x7f15e419f770 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f15e4103170 0x7f15e4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 --2- 172.21.15.26:0/440615745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15e4101f80 0x7f15e41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 >> 172.21.15.26:0/440615745 conn(0x7f15e40fd510 msgr2=0x7f15e4106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 shutdown_connections 2024-04-23T18:16:32.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.606+0000 7f15e8908640 1 -- 172.21.15.26:0/440615745 wait complete. 2024-04-23T18:16:32.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 -- 172.21.15.26:0/1977053216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe348101fe0 msgr2=0x7fe348102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 --2- 172.21.15.26:0/1977053216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe348101fe0 0x7fe348102460 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009900 tx=0x7fe33c02a3f0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 -- 172.21.15.26:0/1977053216 shutdown_connections 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 --2- 172.21.15.26:0/1977053216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3481029a0 0x7fe3481092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 --2- 172.21.15.26:0/1977053216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe348101fe0 0x7fe348102460 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 --2- 172.21.15.26:0/1977053216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348100e90 0x7fe3481012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.721+0000 7fe34f59b640 1 -- 172.21.15.26:0/1977053216 >> 172.21.15.26:0/1977053216 conn(0x7fe3480fc400 msgr2=0x7fe3480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- 172.21.15.26:0/1977053216 shutdown_connections 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- 172.21.15.26:0/1977053216 wait complete. 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 Processor -- start 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- start start 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 0x7fe348069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 0x7fe34819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810c9d0 con 0x7fe3481029a0 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810c850 con 0x7fe348101fe0 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34f59b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe34810cb50 con 0x7fe348100e90 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34db38640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34db38640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38508/0 (socket says 172.21.15.26:38508) 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.722+0000 7fe34db38640 1 -- 172.21.15.26:0/3637691513 learned_addr learned my addr 172.21.15.26:0/3637691513 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34d337640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 0x7fe348069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34cb36640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 0x7fe34819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 -- 172.21.15.26:0/3637691513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 msgr2=0x7fe348069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 0x7fe348069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 -- 172.21.15.26:0/3637691513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 msgr2=0x7fe34819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 0x7fe34819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 -- 172.21.15.26:0/3637691513 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34819fd40 con 0x7fe3481029a0 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34cb36640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 0x7fe34819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34d337640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 0x7fe348069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34db38640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe33800cce0 tx=0x7fe33800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33800f450 con 0x7fe3481029a0 2024-04-23T18:16:32.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe33800faa0 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3481a0030 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338018b70 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.723+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3481a0500 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.724+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3481081f0 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.725+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe338018d10 con 0x7fe3481029a0 2024-04-23T18:16:32.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.726+0000 7fe3467fc640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 0x7fe3300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.726+0000 7fe34d337640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 0x7fe3300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.726+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe3380d0b30 con 0x7fe3481029a0 2024-04-23T18:16:32.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.726+0000 7fe34d337640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 0x7fe3300a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe34000abe0 tx=0x7fe34000a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.729+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe338014030 con 0x7fe3481029a0 2024-04-23T18:16:32.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.916+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe34802d770 con 0x7fe3481029a0 2024-04-23T18:16:32.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.916+0000 7fe3467fc640 1 -- 172.21.15.26:0/3637691513 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fe33809d3f0 con 0x7fe3481029a0 2024-04-23T18:16:32.910 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 msgr2=0x7fe3300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 0x7fe3300a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe34000abe0 tx=0x7fe34000a5c0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 msgr2=0x7fe34819f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe33800cce0 tx=0x7fe33800a620 comp rx=0 tx=0).stop 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 shutdown_connections 2024-04-23T18:16:32.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3300a3c40 0x7fe3300a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe348101fe0 0x7fe34819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3481029a0 0x7fe34819f480 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 --2- 172.21.15.26:0/3637691513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe348100e90 0x7fe348069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.918+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 >> 172.21.15.26:0/3637691513 conn(0x7fe3480fc400 msgr2=0x7fe3481053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.919+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 shutdown_connections 2024-04-23T18:16:32.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:32.919+0000 7fe34f59b640 1 -- 172.21.15.26:0/3637691513 wait complete. 2024-04-23T18:16:32.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 -- 172.21.15.26:0/420098884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 msgr2=0x7f1c1810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 --2- 172.21.15.26:0/420098884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c1810a400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009ca0 tx=0x7f1c1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 -- 172.21.15.26:0/420098884 shutdown_connections 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 --2- 172.21.15.26:0/420098884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c181031b0 0x7f1c18103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 --2- 172.21.15.26:0/420098884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c1810a400 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 --2- 172.21.15.26:0/420098884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c18101fc0 0x7f1c181023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 -- 172.21.15.26:0/420098884 >> 172.21.15.26:0/420098884 conn(0x7f1c180fd550 msgr2=0x7f1c180ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 -- 172.21.15.26:0/420098884 shutdown_connections 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 -- 172.21.15.26:0/420098884 wait complete. 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.032+0000 7f1c2048a640 1 Processor -- start 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- start start 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 0x7f1c1819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c181031b0 0x7f1c1819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1810db60 con 0x7f1c18103b70 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1810d9e0 con 0x7f1c18101fc0 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1810dce0 con 0x7f1c181031b0 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38528/0 (socket says 172.21.15.26:38528) 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 -- 172.21.15.26:0/2335137742 learned_addr learned my addr 172.21.15.26:0/2335137742 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1da25640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c181031b0 0x7f1c1819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1e226640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 0x7f1c1819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 -- 172.21.15.26:0/2335137742 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c181031b0 msgr2=0x7f1c1819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c181031b0 0x7f1c1819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 -- 172.21.15.26:0/2335137742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 msgr2=0x7f1c1819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 0x7f1c1819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 -- 172.21.15.26:0/2335137742 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c181a2070 con 0x7f1c18103b70 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1e226640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 0x7f1c1819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c1ea27640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1c100097b0 tx=0x7f1c10003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10004240 con 0x7f1c18103b70 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c100317c0 con 0x7f1c18103b70 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c181a2300 con 0x7f1c18103b70 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1002fd80 con 0x7f1c18103b70 2024-04-23T18:16:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.033+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c181a28d0 con 0x7f1c18103b70 2024-04-23T18:16:33.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.034+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c181a2b30 con 0x7f1c18103b70 2024-04-23T18:16:33.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.036+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1c100043e0 con 0x7f1c18103b70 2024-04-23T18:16:33.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.036+0000 7f1c0f7fe640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 0x7f1c040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.036+0000 7f1c1e226640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 0x7f1c040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.036+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1c100f1fd0 con 0x7f1c18103b70 2024-04-23T18:16:33.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.037+0000 7f1c1e226640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 0x7f1c040a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.040+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c100bea90 con 0x7f1c18103b70 2024-04-23T18:16:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:32 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/870600061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:32 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/440615745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:32 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3637691513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.267+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c181a2e20 con 0x7f1c18103b70 2024-04-23T18:16:33.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.267+0000 7f1c0f7fe640 1 -- 172.21.15.26:0/2335137742 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f1c100be390 con 0x7f1c18103b70 2024-04-23T18:16:33.261 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:33.261 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:16:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 msgr2=0x7f1c040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 0x7f1c040a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14008040 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 msgr2=0x7f1c181a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1c100097b0 tx=0x7f1c10003c20 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 shutdown_connections 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1c040a3c40 0x7f1c040a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1c18103b70 0x7f1c181a1a00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c181031b0 0x7f1c1819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 --2- 172.21.15.26:0/2335137742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c18101fc0 0x7f1c1819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 >> 172.21.15.26:0/2335137742 conn(0x7f1c180fd550 msgr2=0x7f1c18074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 shutdown_connections 2024-04-23T18:16:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.270+0000 7f1c2048a640 1 -- 172.21.15.26:0/2335137742 wait complete. 2024-04-23T18:16:33.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 -- 172.21.15.26:0/2295599162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568101fc0 msgr2=0x7fa5681023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/2295599162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568101fc0 0x7fa5681023e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa558009900 tx=0x7fa55802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 -- 172.21.15.26:0/2295599162 shutdown_connections 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/2295599162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 0x7fa568103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/2295599162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568103b70 0x7fa56810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/2295599162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568101fc0 0x7fa5681023e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 -- 172.21.15.26:0/2295599162 >> 172.21.15.26:0/2295599162 conn(0x7fa5680fd550 msgr2=0x7fa5680ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 -- 172.21.15.26:0/2295599162 shutdown_connections 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.383+0000 7fa56e2ae640 1 -- 172.21.15.26:0/2295599162 wait complete. 2024-04-23T18:16:33.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 Processor -- start 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 -- start start 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568101fc0 0x7fa568075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 0x7fa568074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810dac0 con 0x7fa568103b70 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810d940 con 0x7fa5681031b0 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56e2ae640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810dc40 con 0x7fa568101fc0 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38548/0 (socket says 172.21.15.26:38548) 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 -- 172.21.15.26:0/1932791862 learned_addr learned my addr 172.21.15.26:0/1932791862 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa5677fe640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 0x7fa568074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa567fff640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568101fc0 0x7fa568075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 -- 172.21.15.26:0/1932791862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568101fc0 msgr2=0x7fa568075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568101fc0 0x7fa568075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 -- 172.21.15.26:0/1932791862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 msgr2=0x7fa568074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.384+0000 7fa56c84b640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 0x7fa568074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa56c84b640 1 -- 172.21.15.26:0/1932791862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa568075110 con 0x7fa568103b70 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa56c84b640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cad0 tx=0x7fa56000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56000ae20 con 0x7fa568103b70 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa560005020 con 0x7fa568103b70 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5681a2360 con 0x7fa568103b70 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56000f6b0 con 0x7fa568103b70 2024-04-23T18:16:33.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.385+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5681a2860 con 0x7fa568103b70 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.386+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5681093b0 con 0x7fa568103b70 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.387+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa5600051d0 con 0x7fa568103b70 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.388+0000 7fa5657fa640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 0x7fa5500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.388+0000 7fa567fff640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 0x7fa5500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.388+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa5600d0750 con 0x7fa568103b70 2024-04-23T18:16:33.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.389+0000 7fa567fff640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 0x7fa5500a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa558000c00 tx=0x7fa5580314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.393+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa560014030 con 0x7fa568103b70 2024-04-23T18:16:33.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:32 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/870600061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:32 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/440615745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:32 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3637691513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:32 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/870600061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:32 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/440615745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:32 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3637691513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.581+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa568062b50 con 0x7fa568103b70 2024-04-23T18:16:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.582+0000 7fa5657fa640 1 -- 172.21.15.26:0/1932791862 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7fa56009cf80 con 0x7fa568103b70 2024-04-23T18:16:33.576 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 msgr2=0x7fa5500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 0x7fa5500a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa558000c00 tx=0x7fa5580314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 msgr2=0x7fa568074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cad0 tx=0x7fa56000cf90 comp rx=0 tx=0).stop 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 shutdown_connections 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa5500a3c40 0x7fa5500a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5681031b0 0x7fa568074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa568103b70 0x7fa568074880 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 --2- 172.21.15.26:0/1932791862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa568101fc0 0x7fa568075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 >> 172.21.15.26:0/1932791862 conn(0x7fa5680fd550 msgr2=0x7fa568106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 shutdown_connections 2024-04-23T18:16:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.584+0000 7fa56e2ae640 1 -- 172.21.15.26:0/1932791862 wait complete. 2024-04-23T18:16:33.589 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:16:33.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 -- 172.21.15.26:0/2895596760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c101fe0 msgr2=0x7f520c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 --2- 172.21.15.26:0/2895596760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c101fe0 0x7f520c102460 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009900 tx=0x7f51fc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 -- 172.21.15.26:0/2895596760 shutdown_connections 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 --2- 172.21.15.26:0/2895596760 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c1029a0 0x7f520c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 --2- 172.21.15.26:0/2895596760 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c101fe0 0x7f520c102460 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 --2- 172.21.15.26:0/2895596760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 0x7f520c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 -- 172.21.15.26:0/2895596760 >> 172.21.15.26:0/2895596760 conn(0x7f520c0fc400 msgr2=0x7f520c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 -- 172.21.15.26:0/2895596760 shutdown_connections 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.702+0000 7f5213dcd640 1 -- 172.21.15.26:0/2895596760 wait complete. 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 Processor -- start 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 -- start start 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 0x7f520c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 0x7f520c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10c9e0 con 0x7f520c1029a0 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10c860 con 0x7f520c100e90 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5213dcd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10cb60 con 0x7f520c101fe0 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38568/0 (socket says 172.21.15.26:38568) 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 -- 172.21.15.26:0/571019850 learned_addr learned my addr 172.21.15.26:0/571019850 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5211368640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 0x7f520c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5211b69640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 0x7f520c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 -- 172.21.15.26:0/571019850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 msgr2=0x7f520c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 0x7f520c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 -- 172.21.15.26:0/571019850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 msgr2=0x7f520c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 0x7f520c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f521236a640 1 -- 172.21.15.26:0/571019850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c19fe50 con 0x7f520c1029a0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.703+0000 7f5211368640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 0x7f520c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f521236a640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f520400eb10 tx=0x7f520400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204004b20 con 0x7f520c1029a0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c1a0140 con 0x7f520c1029a0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5204004cc0 con 0x7f520c1029a0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204010690 con 0x7f520c1029a0 2024-04-23T18:16:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.704+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1a06b0 con 0x7f520c1029a0 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.705+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c1a0970 con 0x7f520c1029a0 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.707+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5204005000 con 0x7f520c1029a0 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.707+0000 7f5202ffd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 0x7f51f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.707+0000 7f5211b69640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 0x7f51f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.707+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f52040d0dc0 con 0x7f520c1029a0 2024-04-23T18:16:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.708+0000 7f5211b69640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 0x7f51f80a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5208005d00 tx=0x7f5208005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.711+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5204053ff0 con 0x7f520c1029a0 2024-04-23T18:16:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.898+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f520c1a0b40 con 0x7f520c1029a0 2024-04-23T18:16:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.898+0000 7f5202ffd640 1 -- 172.21.15.26:0/571019850 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f520409d650 con 0x7f520c1029a0 2024-04-23T18:16:33.892 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:16:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 msgr2=0x7f51f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 0x7f51f80a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5208005d00 tx=0x7f5208005c50 comp rx=0 tx=0).stop 2024-04-23T18:16:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 msgr2=0x7f520c19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f520400eb10 tx=0x7f520400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 shutdown_connections 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f51f80a3c40 0x7f51f80a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f520c1029a0 0x7f520c19f7e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f520c101fe0 0x7f520c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 --2- 172.21.15.26:0/571019850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f520c100e90 0x7f520c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 >> 172.21.15.26:0/571019850 conn(0x7f520c0fc400 msgr2=0x7f520c0690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 shutdown_connections 2024-04-23T18:16:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:33.900+0000 7f5213dcd640 1 -- 172.21.15.26:0/571019850 wait complete. 2024-04-23T18:16:33.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 -- 172.21.15.26:0/531119064 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 msgr2=0x7f1b7c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 --2- 172.21.15.26:0/531119064 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c107770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009900 tx=0x7f1b6c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 -- 172.21.15.26:0/531119064 shutdown_connections 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 --2- 172.21.15.26:0/531119064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c107cb0 0x7f1b7c10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 --2- 172.21.15.26:0/531119064 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c107770 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 --2- 172.21.15.26:0/531119064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c102a50 0x7f1b7c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 -- 172.21.15.26:0/531119064 >> 172.21.15.26:0/531119064 conn(0x7f1b7c0fc3e0 msgr2=0x7f1b7c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 -- 172.21.15.26:0/531119064 shutdown_connections 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.017+0000 7f1b8457d640 1 -- 172.21.15.26:0/531119064 wait complete. 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 Processor -- start 2024-04-23T18:16:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 -- start start 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c102a50 0x7f1b7c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c107cb0 0x7f1b7c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c10da80 con 0x7f1b7c105380 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c10d900 con 0x7f1b7c107cb0 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b8457d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c10dc00 con 0x7f1b7c102a50 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38582/0 (socket says 172.21.15.26:38582) 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 -- 172.21.15.26:0/1718804921 learned_addr learned my addr 172.21.15.26:0/1718804921 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b82b1a640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c107cb0 0x7f1b7c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b82319640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c102a50 0x7f1b7c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 -- 172.21.15.26:0/1718804921 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c102a50 msgr2=0x7f1b7c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c102a50 0x7f1b7c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 -- 172.21.15.26:0/1718804921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c107cb0 msgr2=0x7f1b7c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.018+0000 7f1b81b18640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c107cb0 0x7f1b7c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b81b18640 1 -- 172.21.15.26:0/1718804921 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c19fe00 con 0x7f1b7c105380 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b81b18640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c006010 tx=0x7f1b6c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c03f030 con 0x7f1b7c105380 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b6c003cb0 con 0x7f1b7c105380 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7c1a0090 con 0x7f1b7c105380 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c035070 con 0x7f1b7c105380 2024-04-23T18:16:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.019+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7c1a0580 con 0x7f1b7c105380 2024-04-23T18:16:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.020+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7c0fded0 con 0x7f1b7c105380 2024-04-23T18:16:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.021+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1b6c042030 con 0x7f1b7c105380 2024-04-23T18:16:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.022+0000 7f1b737fe640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 0x7f1b680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.022+0000 7f1b82319640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 0x7f1b680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.022+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1b6c0f0b70 con 0x7f1b7c105380 2024-04-23T18:16:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.022+0000 7f1b82319640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 0x7f1b680a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800bbe0 tx=0x7f1b7800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.027+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b6c0bcd50 con 0x7f1b7c105380 2024-04-23T18:16:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.211+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1b7c02d7e0 con 0x7f1b7c105380 2024-04-23T18:16:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.211+0000 7f1b737fe640 1 -- 172.21.15.26:0/1718804921 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f1b6c0c1050 con 0x7f1b7c105380 2024-04-23T18:16:34.205 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 msgr2=0x7f1b680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 0x7f1b680a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800bbe0 tx=0x7f1b7800b390 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 msgr2=0x7f1b7c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c006010 tx=0x7f1b6c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 shutdown_connections 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1b680a3c40 0x7f1b680a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b7c107cb0 0x7f1b7c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b7c105380 0x7f1b7c198720 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 --2- 172.21.15.26:0/1718804921 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b7c102a50 0x7f1b7c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 >> 172.21.15.26:0/1718804921 conn(0x7f1b7c0fc3e0 msgr2=0x7f1b7c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 shutdown_connections 2024-04-23T18:16:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.213+0000 7f1b8457d640 1 -- 172.21.15.26:0/1718804921 wait complete. 2024-04-23T18:16:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:34 smithi026 ceph-mon[29364]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:34 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2335137742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:34 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1932791862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:34 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/571019850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:34.218 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:16:34.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 -- 172.21.15.26:0/698456750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 msgr2=0x7ff3d00fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/698456750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d00fff90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009900 tx=0x7ff3cc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 -- 172.21.15.26:0/698456750 shutdown_connections 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/698456750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 0x7ff3d0107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/698456750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 0x7ff3d010a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/698456750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d00fff90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 -- 172.21.15.26:0/698456750 >> 172.21.15.26:0/698456750 conn(0x7ff3d00fb750 msgr2=0x7ff3d00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 -- 172.21.15.26:0/698456750 shutdown_connections 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.329+0000 7ff3d6e79640 1 -- 172.21.15.26:0/698456750 wait complete. 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 Processor -- start 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 -- start start 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 0x7ff3d01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 0x7ff3d019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010dab0 con 0x7ff3d00ffb90 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010d930 con 0x7ff3d0100580 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d6e79640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010dc30 con 0x7ff3d0107ca0 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38594/0 (socket says 172.21.15.26:38594) 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 -- 172.21.15.26:0/240164122 learned_addr learned my addr 172.21.15.26:0/240164122 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 -- 172.21.15.26:0/240164122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 msgr2=0x7ff3d019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d5416640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 0x7ff3d019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 0x7ff3d019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 -- 172.21.15.26:0/240164122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 msgr2=0x7ff3d01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3c7fff640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 0x7ff3d01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 0x7ff3d01986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 -- 172.21.15.26:0/240164122 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d019fd80 con 0x7ff3d00ffb90 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3d4c15640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc005940 tx=0x7ff3cc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.330+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc03f030 con 0x7ff3d00ffb90 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.331+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3cc003cb0 con 0x7ff3d00ffb90 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.331+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d01a0010 con 0x7ff3d00ffb90 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.331+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc035070 con 0x7ff3d00ffb90 2024-04-23T18:16:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.331+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d01a0500 con 0x7ff3d00ffb90 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.332+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d0106570 con 0x7ff3d00ffb90 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.333+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff3cc042030 con 0x7ff3d00ffb90 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.333+0000 7ff3c5ffb640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 0x7ff3b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.333+0000 7ff3c7fff640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 0x7ff3b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.333+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff3cc0f0fb0 con 0x7ff3d00ffb90 2024-04-23T18:16:34.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.334+0000 7ff3c7fff640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 0x7ff3b80a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0199670 tx=0x7ff3c0005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.338+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3cc003e60 con 0x7ff3d00ffb90 2024-04-23T18:16:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:34 smithi110 ceph-mon[32585]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:34 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2335137742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:34 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1932791862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:34 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/571019850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:34.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:34 smithi131 ceph-mon[33041]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:34.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:34 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2335137742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:34 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1932791862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:34.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:34 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/571019850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.530+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff3d0062b50 con 0x7ff3d00ffb90 2024-04-23T18:16:34.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.530+0000 7ff3c5ffb640 1 -- 172.21.15.26:0/240164122 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff3cc0bd840 con 0x7ff3d00ffb90 2024-04-23T18:16:34.524 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:16:34.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 msgr2=0x7ff3b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 0x7ff3b80a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0199670 tx=0x7ff3c0005cc0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 msgr2=0x7ff3d0198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc005940 tx=0x7ff3cc004430 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 shutdown_connections 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7ff3b80a3c40 0x7ff3b80a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d0100580 0x7ff3d01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0107ca0 0x7ff3d019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 --2- 172.21.15.26:0/240164122 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff3d00ffb90 0x7ff3d0198160 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 >> 172.21.15.26:0/240164122 conn(0x7ff3d00fb750 msgr2=0x7ff3d0103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 shutdown_connections 2024-04-23T18:16:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.533+0000 7ff3d6e79640 1 -- 172.21.15.26:0/240164122 wait complete. 2024-04-23T18:16:34.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- 172.21.15.26:0/3519012524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48102040 msgr2=0x7fed481024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 --2- 172.21.15.26:0/3519012524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48102040 0x7fed481024c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009900 tx=0x7fed3c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- 172.21.15.26:0/3519012524 shutdown_connections 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 --2- 172.21.15.26:0/3519012524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 0x7fed48109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 --2- 172.21.15.26:0/3519012524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48102040 0x7fed481024c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 --2- 172.21.15.26:0/3519012524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48100e60 0x7fed48101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- 172.21.15.26:0/3519012524 >> 172.21.15.26:0/3519012524 conn(0x7fed480fc440 msgr2=0x7fed480fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- 172.21.15.26:0/3519012524 shutdown_connections 2024-04-23T18:16:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- 172.21.15.26:0/3519012524 wait complete. 2024-04-23T18:16:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 Processor -- start 2024-04-23T18:16:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.646+0000 7fed4d907640 1 -- start start 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 0x7fed481987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 0x7fed4819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fed4810cac0 con 0x7fed48100e60 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fed4810c940 con 0x7fed48102040 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fed4810cc40 con 0x7fed48102a00 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38624/0 (socket says 172.21.15.26:38624) 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 -- 172.21.15.26:0/2364238148 learned_addr learned my addr 172.21.15.26:0/2364238148 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed477fe640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 0x7fed4819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 -- 172.21.15.26:0/2364238148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 msgr2=0x7fed4819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 0x7fed4819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed467fc640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 0x7fed481987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 -- 172.21.15.26:0/2364238148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 msgr2=0x7fed481987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 0x7fed481987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 -- 172.21.15.26:0/2364238148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4819feb0 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed467fc640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 0x7fed481987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed46ffd640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fed3800b450 tx=0x7fed3800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38015020 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed38011d40 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed481a01a0 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800f340 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.647+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed481a0770 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.648+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed481a0a30 con 0x7fed48100e60 2024-04-23T18:16:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.650+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fed38018020 con 0x7fed48100e60 2024-04-23T18:16:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.650+0000 7fed4c905640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 0x7fed300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.651+0000 7fed467fc640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 0x7fed300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.651+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fed380d0590 con 0x7fed48100e60 2024-04-23T18:16:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.651+0000 7fed467fc640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 0x7fed300a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed3c02a930 tx=0x7fed3c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.654+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed3801f020 con 0x7fed48100e60 2024-04-23T18:16:34.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.880+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed481a0c00 con 0x7fed48100e60 2024-04-23T18:16:34.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.881+0000 7fed4c905640 1 -- 172.21.15.26:0/2364238148 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fed3800d040 con 0x7fed48100e60 2024-04-23T18:16:34.875 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:34.875 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:16:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:16:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 msgr2=0x7fed300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 0x7fed300a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed3c02a930 tx=0x7fed3c005fd0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 msgr2=0x7fed48198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fed3800b450 tx=0x7fed3800b910 comp rx=0 tx=0).stop 2024-04-23T18:16:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 shutdown_connections 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fed300a3c40 0x7fed300a6100 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed48102040 0x7fed481987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed48102a00 0x7fed4819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 --2- 172.21.15.26:0/2364238148 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fed48100e60 0x7fed48198290 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 >> 172.21.15.26:0/2364238148 conn(0x7fed480fc440 msgr2=0x7fed480fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 shutdown_connections 2024-04-23T18:16:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.883+0000 7fed4d907640 1 -- 172.21.15.26:0/2364238148 wait complete. 2024-04-23T18:16:34.888 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 -- 172.21.15.26:0/1065059681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 msgr2=0x7fca701024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 --2- 172.21.15.26:0/1065059681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701024c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fca60009900 tx=0x7fca6002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 -- 172.21.15.26:0/1065059681 shutdown_connections 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 --2- 172.21.15.26:0/1065059681 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70102a00 0x7fca70109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 --2- 172.21.15.26:0/1065059681 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701024c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 --2- 172.21.15.26:0/1065059681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70100e60 0x7fca70101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 -- 172.21.15.26:0/1065059681 >> 172.21.15.26:0/1065059681 conn(0x7fca700fc440 msgr2=0x7fca700fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 -- 172.21.15.26:0/1065059681 shutdown_connections 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.997+0000 7fca78491640 1 -- 172.21.15.26:0/1065059681 wait complete. 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 Processor -- start 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 -- start start 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70100e60 0x7fca70104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70102a00 0x7fca701039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fca7010c9f0 con 0x7fca70102040 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fca7010c870 con 0x7fca70102a00 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca78491640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fca7010cb70 con 0x7fca70100e60 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38646/0 (socket says 172.21.15.26:38646) 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 -- 172.21.15.26:0/1949343210 learned_addr learned my addr 172.21.15.26:0/1949343210 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca76a2e640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70102a00 0x7fca701039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca7622d640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70100e60 0x7fca70104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 -- 172.21.15.26:0/1949343210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70100e60 msgr2=0x7fca70104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70100e60 0x7fca70104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 -- 172.21.15.26:0/1949343210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70102a00 msgr2=0x7fca701039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70102a00 0x7fca701039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.998+0000 7fca75a2c640 1 -- 172.21.15.26:0/1949343210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca70104280 con 0x7fca70102040 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca75a2c640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fca60031500 tx=0x7fca600043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6003f030 con 0x7fca70102040 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca60003800 con 0x7fca70102040 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca701a2220 con 0x7fca70102040 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60035070 con 0x7fca70102040 2024-04-23T18:16:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:34.999+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca701a2770 con 0x7fca70102040 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.000+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca701a2a90 con 0x7fca70102040 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.001+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fca600039b0 con 0x7fca70102040 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.002+0000 7fca677fe640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 0x7fca5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.002+0000 7fca7622d640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 0x7fca5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.002+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fca600f15f0 con 0x7fca70102040 2024-04-23T18:16:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.002+0000 7fca7622d640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 0x7fca5c0a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fca6c005db0 tx=0x7fca6c005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.007+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca600bde80 con 0x7fca70102040 2024-04-23T18:16:35.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.195+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca701a2c60 con 0x7fca70102040 2024-04-23T18:16:35.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.197+0000 7fca677fe640 1 -- 172.21.15.26:0/1949343210 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7fca600bd780 con 0x7fca70102040 2024-04-23T18:16:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:35.191 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 msgr2=0x7fca5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 0x7fca5c0a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fca6c005db0 tx=0x7fca6c005d40 comp rx=0 tx=0).stop 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 msgr2=0x7fca701034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fca60031500 tx=0x7fca600043c0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 shutdown_connections 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fca5c0a3c40 0x7fca5c0a6100 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca70102a00 0x7fca701039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fca70102040 0x7fca701034b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 --2- 172.21.15.26:0/1949343210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca70100e60 0x7fca70104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 >> 172.21.15.26:0/1949343210 conn(0x7fca700fc440 msgr2=0x7fca700fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 shutdown_connections 2024-04-23T18:16:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.199+0000 7fca78491640 1 -- 172.21.15.26:0/1949343210 wait complete. 2024-04-23T18:16:35.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:35 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1718804921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:35 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/240164122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:35 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2364238148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 -- 172.21.15.26:0/3801849669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102040 msgr2=0x7f54e01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/3801849669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102040 0x7f54e01024c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009900 tx=0x7f54d002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 -- 172.21.15.26:0/3801849669 shutdown_connections 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/3801849669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e0102a00 0x7f54e0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/3801849669 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102040 0x7f54e01024c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/3801849669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 0x7f54e0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 -- 172.21.15.26:0/3801849669 >> 172.21.15.26:0/3801849669 conn(0x7f54e00fc440 msgr2=0x7f54e00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 -- 172.21.15.26:0/3801849669 shutdown_connections 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.312+0000 7f54e7c7b640 1 -- 172.21.15.26:0/3801849669 wait complete. 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 Processor -- start 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 -- start start 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 0x7f54e0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e0102040 0x7f54e0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f54e010c9f0 con 0x7f54e0102a00 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54e010c870 con 0x7f54e0100e60 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e7c7b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f54e010cb70 con 0x7f54e0102040 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e6218640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e6218640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38658/0 (socket says 172.21.15.26:38658) 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e6218640 1 -- 172.21.15.26:0/127569420 learned_addr learned my addr 172.21.15.26:0/127569420 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.313+0000 7f54e5a17640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 0x7f54e0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 -- 172.21.15.26:0/127569420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e0102040 msgr2=0x7f54e0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e0102040 0x7f54e0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 -- 172.21.15.26:0/127569420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 msgr2=0x7f54e0198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 0x7f54e0198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 -- 172.21.15.26:0/127569420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e019fe60 con 0x7f54e0102a00 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e6218640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54d800cad0 tx=0x7f54d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d800ae20 con 0x7f54e0102a00 2024-04-23T18:16:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d8005020 con 0x7f54e0102a00 2024-04-23T18:16:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e01a0150 con 0x7f54e0102a00 2024-04-23T18:16:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d800f6b0 con 0x7f54e0102a00 2024-04-23T18:16:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.314+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e01a0650 con 0x7f54e0102a00 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.315+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e01a0910 con 0x7f54e0102a00 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.316+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f54d80051d0 con 0x7f54e0102a00 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.317+0000 7f54d6ffd640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 0x7f54cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.317+0000 7f54e5a17640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 0x7f54cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.317+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f54d80d0640 con 0x7f54e0102a00 2024-04-23T18:16:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.317+0000 7f54e5a17640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 0x7f54cc0a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54e0101bb0 tx=0x7f54dc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.321+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54d8014030 con 0x7f54e0102a00 2024-04-23T18:16:35.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:35 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1718804921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:35 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/240164122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:35 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2364238148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:35 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1718804921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:35 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/240164122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:35.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:35 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2364238148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.549+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54e01a0b80 con 0x7f54e0102a00 2024-04-23T18:16:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.550+0000 7f54d6ffd640 1 -- 172.21.15.26:0/127569420 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f54d809ced0 con 0x7f54e0102a00 2024-04-23T18:16:35.543 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:35.544 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:16:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:16:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 msgr2=0x7f54cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 0x7f54cc0a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54e0101bb0 tx=0x7f54dc00a2d0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 msgr2=0x7f54e019f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54d800cad0 tx=0x7f54d800cf90 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 shutdown_connections 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f54cc0a3c40 0x7f54cc0a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e0102a00 0x7f54e019f7f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e0102040 0x7f54e0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 --2- 172.21.15.26:0/127569420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e0100e60 0x7f54e0198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 >> 172.21.15.26:0/127569420 conn(0x7f54e00fc440 msgr2=0x7f54e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 shutdown_connections 2024-04-23T18:16:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.552+0000 7f54e7c7b640 1 -- 172.21.15.26:0/127569420 wait complete. 2024-04-23T18:16:35.556 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:15:06.048360+0000', 'modified': '2024-04-23T18:15:11.649607+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24464, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24433': {'gid': 24433, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6835/2220145154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6834', 'nonce': 2220145154}, {'type': 'v1', 'addr': '172.21.15.131:6835', 'nonce': 2220145154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/3666168856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 3666168856}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 3666168856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6837/2204987461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6836', 'nonce': 2204987461}, {'type': 'v1', 'addr': '172.21.15.131:6837', 'nonce': 2204987461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.131:6839/4056718895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6838', 'nonce': 4056718895}, {'type': 'v1', 'addr': '172.21.15.131:6839', 'nonce': 4056718895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.110:6837/3430274697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 3430274697}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 3430274697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6839/3619972803', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 3619972803}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 3619972803}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24464, 24503, 24460]} 2024-04-23T18:16:35.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:16:35.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:16:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 -- 172.21.15.26:0/3690743192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 msgr2=0x7fcd681012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 --2- 172.21.15.26:0/3690743192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681012b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009900 tx=0x7fcd5c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 -- 172.21.15.26:0/3690743192 shutdown_connections 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 --2- 172.21.15.26:0/3690743192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd681029a0 0x7fcd681092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 --2- 172.21.15.26:0/3690743192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcd68101fe0 0x7fcd68102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 --2- 172.21.15.26:0/3690743192 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681012b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 -- 172.21.15.26:0/3690743192 >> 172.21.15.26:0/3690743192 conn(0x7fcd680fc400 msgr2=0x7fcd680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 -- 172.21.15.26:0/3690743192 shutdown_connections 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.667+0000 7fcd6f064640 1 -- 172.21.15.26:0/3690743192 wait complete. 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 Processor -- start 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 -- start start 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd68101fe0 0x7fcd681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcd681029a0 0x7fcd6819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810c9e0 con 0x7fcd68100e90 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810c860 con 0x7fcd68101fe0 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6f064640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810cb60 con 0x7fcd681029a0 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38682/0 (socket says 172.21.15.26:38682) 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 -- 172.21.15.26:0/898994286 learned_addr learned my addr 172.21.15.26:0/898994286 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd67fff640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd68101fe0 0x7fcd681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 -- 172.21.15.26:0/898994286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcd681029a0 msgr2=0x7fcd6819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcd681029a0 0x7fcd6819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 -- 172.21.15.26:0/898994286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd68101fe0 msgr2=0x7fcd681986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd68101fe0 0x7fcd681986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.668+0000 7fcd6ce00640 1 -- 172.21.15.26:0/898994286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6819fdd0 con 0x7fcd68100e90 2024-04-23T18:16:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c005940 tx=0x7fcd5c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd65ffb640 1 -- 172.21.15.26:0/898994286 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c03f030 con 0x7fcd68100e90 2024-04-23T18:16:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd65ffb640 1 -- 172.21.15.26:0/898994286 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5c003cb0 con 0x7fcd68100e90 2024-04-23T18:16:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd681a0060 con 0x7fcd68100e90 2024-04-23T18:16:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd65ffb640 1 -- 172.21.15.26:0/898994286 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c035070 con 0x7fcd68100e90 2024-04-23T18:16:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.669+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd681a0550 con 0x7fcd68100e90 2024-04-23T18:16:35.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd65ffb640 1 -- 172.21.15.26:0/898994286 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcd5c042030 con 0x7fcd68100e90 2024-04-23T18:16:35.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 Processor -- start 2024-04-23T18:16:35.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 -- start start 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603826f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x5603826f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560382658c50 con 0x5603826f9b20 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5603825f5830 con 0x5603826f9160 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560382654c10 con 0x5603826f7f90 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38694/0 (socket says 172.21.15.26:38694) 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd67fff640 1 -- 172.21.15.26:0/1868899373 learned_addr learned my addr 172.21.15.26:0/1868899373 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x5603826f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.672+0000 7fcd65ffb640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 0x7fcd580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603826f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 -- 172.21.15.26:0/1868899373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 msgr2=0x5603826f83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603826f83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 -- 172.21.15.26:0/1868899373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 msgr2=0x5603826f95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x5603826f95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 -- 172.21.15.26:0/1868899373 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56038236b2d0 con 0x5603826f9b20 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd65ffb640 1 -- 172.21.15.26:0/898994286 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcd5c0f0fe0 con 0x7fcd68100e90 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd6d601640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 0x7fcd580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009b20 tx=0x7fcd5400e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf37d2dbf70ceca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd52ffd640 1 -- 172.21.15.26:0/1868899373 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54016070 con 0x5603826f9b20 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd52ffd640 1 -- 172.21.15.26:0/1868899373 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5400ed20 con 0x5603826f9b20 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.673+0000 7fcd52ffd640 1 -- 172.21.15.26:0/1868899373 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54011470 con 0x5603826f9b20 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1868899373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 msgr2=0x5603827003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd6d601640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 0x7fcd580a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5603826f9060 tx=0x7fcd60004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009b20 tx=0x7fcd5400e2f0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1868899373 shutdown_connections 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x5603826f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827003b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1868899373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603826f83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1868899373 >> 172.21.15.26:0/1868899373 conn(0x5603826ef070 msgr2=0x5603826ef470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1868899373 shutdown_connections 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1868899373 wait complete. 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 -- start start 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.674+0000 7fcd71d7b740 1 Processor -- start 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x560382702470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560382700b30 con 0x5603826f9b20 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5603825f3470 con 0x5603826f7f90 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560382700cb0 con 0x5603826f9160 2024-04-23T18:16:35.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 10 client.0 _ll_register_callbacks cb 0x560382618f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38696/0 (socket says 172.21.15.26:38696) 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 -- 172.21.15.26:0/756190524 learned_addr learned my addr 172.21.15.26:0/756190524 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd67fff640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 -- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 msgr2=0x5603827029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 -- 172.21.15.26:0/756190524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 msgr2=0x560382702470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x560382702470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 -- 172.21.15.26:0/756190524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603824356b0 con 0x5603826f9b20 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd67fff640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd6d601640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcd60011df0 tx=0x7fcd60007450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71facdc249f9645 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd47fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60022020 con 0x5603826f9b20 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcd6001e450 con 0x5603826f9b20 2024-04-23T18:16:35.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd5400ed20 con 0x5603826f9b20 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd6001e650 con 0x5603826f9b20 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 10 client.15069 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.675+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56038270cd50 con 0x5603826f9b20 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5400ed20 con 0x5603826f9b20 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcd6000f040 con 0x5603826f9b20 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 10 client.15069 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 20 client.15069 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x56038270f500 0x560382710970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 4 client.15069 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560382666670 con 0x56038270f500 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.676+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x56038270f500 0x560382710970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x56038270f500 0x560382710970 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a70856298c82dd05 server_cookie=50099e738ede4095 in_seq=0 out_seq=0 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd47fff640 10 client.15069 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:16:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd47fff640 1 -- 172.21.15.26:0/756190524 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560382666670 con 0x56038270f500 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd47fff640 10 client.15069 handle_command_reply: tid=0 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd71d7b740 1 client.15069 shutdown 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 msgr2=0x560382709910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.677+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcd60011df0 tx=0x7fcd60007450 comp rx=0 tx=0).stop 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 shutdown_connections 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x56038270f500 0x560382710970 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382709910 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/756190524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x560382702470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 >> 172.21.15.26:0/756190524 conn(0x5603826ef070 msgr2=0x5603826ee1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd47fff640 0 client.15069 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 shutdown_connections 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 1 -- 172.21.15.26:0/756190524 wait complete. 2024-04-23T18:16:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.678+0000 7fcd71d7b740 20 client.15069 trim_cache size 0 max 16384 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 Processor -- start 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 -- start start 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x5603826ef450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56038270ecb0 con 0x5603826f9b20 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56038270f380 con 0x5603826f7f90 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56038270fa50 con 0x5603826f9160 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.696+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38712/0 (socket says 172.21.15.26:38712) 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 -- 172.21.15.26:0/1562086287 learned_addr learned my addr 172.21.15.26:0/1562086287 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x5603826ef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 -- 172.21.15.26:0/1562086287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 msgr2=0x5603827098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 -- 172.21.15.26:0/1562086287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 msgr2=0x5603826ef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x5603826ef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 -- 172.21.15.26:0/1562086287 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603826ea720 con 0x5603826f9b20 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcd60005fd0 tx=0x7fcd6001e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa4e05d1278cedd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd527fc640 1 -- 172.21.15.26:0/1562086287 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60039070 con 0x5603826f9b20 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd527fc640 1 -- 172.21.15.26:0/1562086287 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd60007db0 con 0x5603826f9b20 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.697+0000 7fcd527fc640 1 -- 172.21.15.26:0/1562086287 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60022040 con 0x5603826f9b20 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1562086287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 msgr2=0x560382702470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcd60005fd0 tx=0x7fcd6001e550 comp rx=0 tx=0).stop 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1562086287 shutdown_connections 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x560382702470 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f9160 0x5603827098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1562086287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f7f90 0x5603826ef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1562086287 >> 172.21.15.26:0/1562086287 conn(0x5603826ef070 msgr2=0x56038270e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1562086287 shutdown_connections 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1562086287 wait complete. 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- start start 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 Processor -- start 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603827078c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x560382771490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5603827107f0 con 0x5603826f9b20 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560382710120 con 0x5603826f9160 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560382710ec0 con 0x5603826f7f90 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd71d7b740 10 client.0 _ll_register_callbacks cb 0x7fcd6000af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.698+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38722/0 (socket says 172.21.15.26:38722) 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 -- 172.21.15.26:0/1875654317 learned_addr learned my addr 172.21.15.26:0/1875654317 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x560382771490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603827078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 -- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 msgr2=0x5603827078c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603827078c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 -- 172.21.15.26:0/1875654317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 msgr2=0x560382771490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x560382771490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 -- 172.21.15.26:0/1875654317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560382773f20 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd67fff640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x560382771490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd6d601640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcd60014040 tx=0x7fcd60007640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a33da10bb9428db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd47fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60039040 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcd60004d20 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd60007db0 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd60022040 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd71d7b740 10 client.15075 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.699+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560382775100 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd60007db0 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcd6003c070 con 0x5603826f9b20 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 10 client.15075 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:16:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 20 client.15075 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x560382779a70 0x560382778780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 4 client.15075 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "status"}) -- 0x5603826604e0 con 0x560382779a70 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.700+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x560382779a70 0x560382778780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd47fff640 10 client.15075 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd6ce00640 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x560382779a70 0x560382778780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5665167db3bc59a4 server_cookie=a116d4423f39d381 in_seq=0 out_seq=0 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd47fff640 1 -- 172.21.15.26:0/1875654317 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5603826604e0 con 0x560382779a70 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd47fff640 10 client.15075 handle_command_reply: tid=0 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 client.15075 shutdown 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 msgr2=0x5603827739e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcd60014040 tx=0x7fcd60007640 comp rx=0 tx=0).stop 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 shutdown_connections 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x560382779a70 0x560382778780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5603826f9160 0x560382771490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5603826f9b20 0x5603827739e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 --2- 172.21.15.26:0/1875654317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5603826f7f90 0x5603827078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd47fff640 0 client.15075 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 >> 172.21.15.26:0/1875654317 conn(0x5603826ef070 msgr2=0x560382702470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 shutdown_connections 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 1 -- 172.21.15.26:0/1875654317 wait complete. 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.701+0000 7fcd71d7b740 20 client.15075 trim_cache size 0 max 16384 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 2, 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24460, 2024-04-23T18:16:35.702 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 85.002324595999994, 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 103.599561241 2024-04-23T18:16:35.703 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 msgr2=0x7fcd580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 0x7fcd580a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5603826f9060 tx=0x7fcd60004a40 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 msgr2=0x7fcd681981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c005940 tx=0x7fcd5c004430 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 shutdown_connections 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fcd580a3c40 0x7fcd580a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd68101fe0 0x7fcd681986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcd681029a0 0x7fcd6819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 --2- 172.21.15.26:0/898994286 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcd68100e90 0x7fcd681981b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 >> 172.21.15.26:0/898994286 conn(0x7fcd680fc400 msgr2=0x7fcd680fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 shutdown_connections 2024-04-23T18:16:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.702+0000 7fcd6f064640 1 -- 172.21.15.26:0/898994286 wait complete. 2024-04-23T18:16:35.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 -- 172.21.15.26:0/2563825110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 msgr2=0x7f0d04104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/2563825110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d04104e00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009900 tx=0x7f0cfc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 -- 172.21.15.26:0/2563825110 shutdown_connections 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/2563825110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04107c70 0x7f0d0410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/2563825110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04105340 0x7f0d04107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/2563825110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d04104e00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 -- 172.21.15.26:0/2563825110 >> 172.21.15.26:0/2563825110 conn(0x7f0d040fc400 msgr2=0x7f0d040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 -- 172.21.15.26:0/2563825110 shutdown_connections 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.815+0000 7f0d0ba79640 1 -- 172.21.15.26:0/2563825110 wait complete. 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 Processor -- start 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 -- start start 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 0x7f0d04198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04107c70 0x7f0d0419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0410db20 con 0x7f0d04102a10 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0410d9a0 con 0x7f0d04105340 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0ba79640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0410dca0 con 0x7f0d04107c70 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d09815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d09815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38742/0 (socket says 172.21.15.26:38742) 2024-04-23T18:16:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d09815640 1 -- 172.21.15.26:0/3730637124 learned_addr learned my addr 172.21.15.26:0/3730637124 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d0a016640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04107c70 0x7f0d0419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.816+0000 7f0d09014640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 0x7f0d04198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 -- 172.21.15.26:0/3730637124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04107c70 msgr2=0x7f0d0419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04107c70 0x7f0d0419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 -- 172.21.15.26:0/3730637124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 msgr2=0x7f0d04198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 0x7f0d04198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 -- 172.21.15.26:0/3730637124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0419fe10 con 0x7f0d04102a10 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09014640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 0x7f0d04198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d09815640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005ab0 tx=0x7f0cfc003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d02ffd640 1 -- 172.21.15.26:0/3730637124 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc03f070 con 0x7f0d04102a10 2024-04-23T18:16:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d02ffd640 1 -- 172.21.15.26:0/3730637124 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cfc03b460 con 0x7f0d04102a10 2024-04-23T18:16:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d041a00a0 con 0x7f0d04102a10 2024-04-23T18:16:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d02ffd640 1 -- 172.21.15.26:0/3730637124 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc035030 con 0x7f0d04102a10 2024-04-23T18:16:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.817+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d041a0590 con 0x7f0d04102a10 2024-04-23T18:16:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.819+0000 7f0d02ffd640 1 -- 172.21.15.26:0/3730637124 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0cfc042070 con 0x7f0d04102a10 2024-04-23T18:16:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.819+0000 7f0d0e78e740 1 Processor -- start 2024-04-23T18:16:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 -- start start 2024-04-23T18:16:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced88d50 0x5646ced891d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced8ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5646cececc50 con 0x5646ced87b80 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646cec89830 con 0x5646ced88d50 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5646cece8c10 con 0x5646ced89710 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38756/0 (socket says 172.21.15.26:38756) 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 -- 172.21.15.26:0/2975553592 learned_addr learned my addr 172.21.15.26:0/2975553592 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d02ffd640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 0x7f0cec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d09014640 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced8ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d09815640 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced88d50 0x5646ced891d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 -- 172.21.15.26:0/2975553592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 msgr2=0x5646ced8ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced8ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 -- 172.21.15.26:0/2975553592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced88d50 msgr2=0x5646ced891d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced88d50 0x5646ced891d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 -- 172.21.15.26:0/2975553592 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ce9ff2d0 con 0x5646ced87b80 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d02ffd640 1 -- 172.21.15.26:0/3730637124 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0cfc0f0ee0 con 0x7f0d04102a10 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.820+0000 7f0d0a016640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 0x7f0cec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0d0a016640 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010510 tx=0x7f0cf4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e66e84f72a0366 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0cebfff640 1 -- 172.21.15.26:0/2975553592 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4023070 con 0x5646ced87b80 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0cebfff640 1 -- 172.21.15.26:0/2975553592 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf40134c0 con 0x5646ced87b80 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0cebfff640 1 -- 172.21.15.26:0/2975553592 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40117e0 con 0x5646ced87b80 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2975553592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 msgr2=0x5646ced87fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0d0a016640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 0x7f0cec0a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4020000 tx=0x7f0cf401b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.821+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010510 tx=0x7f0cf4009f90 comp rx=0 tx=0).stop 2024-04-23T18:16:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2975553592 shutdown_connections 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced88d50 0x5646ced891d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced8ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2975553592 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced87fa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2975553592 >> 172.21.15.26:0/2975553592 conn(0x5646ced83070 msgr2=0x5646ced83470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2975553592 shutdown_connections 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2975553592 wait complete. 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- start start 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 Processor -- start 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced92090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced995a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5646ced90720 con 0x5646ced88d50 2024-04-23T18:16:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646cec87470 con 0x5646ced87b80 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.822+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5646ced908a0 con 0x5646ced89710 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0e78e740 10 client.0 _ll_register_callbacks cb 0x5646cecacf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38768/0 (socket says 172.21.15.26:38768) 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 -- 172.21.15.26:0/2224138566 learned_addr learned my addr 172.21.15.26:0/2224138566 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0a016640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09815640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced92090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 -- 172.21.15.26:0/2224138566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 msgr2=0x5646ced995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 -- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 msgr2=0x5646ced92090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced92090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 -- 172.21.15.26:0/2224138566 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ceac96b0 con 0x5646ced88d50 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0a016640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09815640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced92090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d09014640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800be90 tx=0x7f0cf80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1eadbc5dc983c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0ce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8005220 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0cf80053c0 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf40134c0 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0cf80055c0 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0e78e740 10 client.15087 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.823+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5646ced9c9e0 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf40134c0 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0cf801b930 con 0x5646ced88d50 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 10 client.15087 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 10 client.15087 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 20 client.15087 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646ced9f0a0 0x5646ceda0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 4 client.15087 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646cecfa670 con 0x5646ced9f0a0 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d09815640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646ced9f0a0 0x5646ceda0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0ce97fa640 10 client.15087 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.824+0000 7f0d09815640 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646ced9f0a0 0x5646ceda0510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4706f7b998c8d793 server_cookie=f551dce7cdb9a92 in_seq=0 out_seq=0 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.825+0000 7f0ce97fa640 1 -- 172.21.15.26:0/2224138566 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646cecfa670 con 0x5646ced9f0a0 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.825+0000 7f0ce97fa640 10 client.15087 handle_command_reply: tid=0 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.825+0000 7f0d0e78e740 1 client.15087 shutdown 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 msgr2=0x5646ced925d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800be90 tx=0x7f0cf80117b0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 shutdown_connections 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646ced9f0a0 0x5646ceda0510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced89710 0x5646ced995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced88d50 0x5646ced925d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/2224138566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced92090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0ce97fa640 0 client.15087 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 >> 172.21.15.26:0/2224138566 conn(0x5646ced83070 msgr2=0x5646ced821d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 shutdown_connections 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 1 -- 172.21.15.26:0/2224138566 wait complete. 2024-04-23T18:16:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.826+0000 7f0d0e78e740 20 client.15087 trim_cache size 0 max 16384 2024-04-23T18:16:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 Processor -- start 2024-04-23T18:16:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 -- start start 2024-04-23T18:16:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced8bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced99560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5646ced9e850 con 0x5646ced89710 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646ced9ef20 con 0x5646ced87b80 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5646ced9f5f0 con 0x5646ced88d50 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.844+0000 7f0d0a016640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38782/0 (socket says 172.21.15.26:38782) 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 -- 172.21.15.26:0/408783530 learned_addr learned my addr 172.21.15.26:0/408783530 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d09815640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced8bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 -- 172.21.15.26:0/408783530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 msgr2=0x5646ced99560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d09014640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced99560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced99560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 -- 172.21.15.26:0/408783530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 msgr2=0x5646ced8bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced8bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 -- 172.21.15.26:0/408783530 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ced887d0 con 0x5646ced89710 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d09815640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced8bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0d0a016640 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009aa0 tx=0x7f0cf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ac1c1afe256092 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0ceb7fe640 1 -- 172.21.15.26:0/408783530 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4023070 con 0x5646ced89710 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0ceb7fe640 1 -- 172.21.15.26:0/408783530 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf40113e0 con 0x5646ced89710 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.845+0000 7f0ceb7fe640 1 -- 172.21.15.26:0/408783530 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40050e0 con 0x5646ced89710 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- 172.21.15.26:0/408783530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 msgr2=0x5646ced7e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009aa0 tx=0x7f0cf4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- 172.21.15.26:0/408783530 shutdown_connections 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced89710 0x5646ced7e0b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced99560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/408783530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced87b80 0x5646ced8bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- 172.21.15.26:0/408783530 >> 172.21.15.26:0/408783530 conn(0x5646ced83070 msgr2=0x5646ced83450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- 172.21.15.26:0/408783530 shutdown_connections 2024-04-23T18:16:35.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- 172.21.15.26:0/408783530 wait complete. 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 -- start start 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.846+0000 7f0d0e78e740 1 Processor -- start 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced94db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 0x5646ced97d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5646ceda0390 con 0x5646ced87b80 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646ced9fcc0 con 0x5646ced89710 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5646ceda0a60 con 0x5646ced88d50 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 10 client.0 _ll_register_callbacks cb 0x7f0cf800ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38786/0 (socket says 172.21.15.26:38786) 2024-04-23T18:16:35.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 -- 172.21.15.26:0/3493648954 learned_addr learned my addr 172.21.15.26:0/3493648954 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09014640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced94db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0a016640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 0x5646ced97d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 -- 172.21.15.26:0/3493648954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 msgr2=0x5646ced94db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced94db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 -- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 msgr2=0x5646ced97d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 0x5646ced97d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 -- 172.21.15.26:0/3493648954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646ced98350 con 0x5646ced87b80 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09014640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced94db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0a016640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 0x5646ced97d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d09815640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0bcfa0 tx=0x7f0cfc03bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc598960e331bb24 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0ce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc049040 con 0x5646ced87b80 2024-04-23T18:16:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0cfc03b610 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf40113e0 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0cfc0bd010 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 10 client.15093 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.847+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5646ced99650 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf40113e0 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0cfc03b8d0 con 0x5646ced87b80 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 10 client.15093 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 20 client.15093 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646cee098d0 0x5646cee085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 4 client.15093 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "status"}) -- 0x5646cecf44e0 con 0x5646cee098d0 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.848+0000 7f0d09014640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646cee098d0 0x5646cee085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d09014640 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646cee098d0 0x5646cee085e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=92f6a57cb1fcedc7 server_cookie=69bc2f33dc8ef5bc in_seq=0 out_seq=0 2024-04-23T18:16:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0ce97fa640 10 client.15093 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0ce97fa640 1 -- 172.21.15.26:0/3493648954 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5646cecf44e0 con 0x5646cee098d0 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0ce97fa640 10 client.15093 handle_command_reply: tid=0 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 client.15093 shutdown 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 msgr2=0x5646ced94870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0bcfa0 tx=0x7f0cfc03bc70 comp rx=0 tx=0).stop 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 shutdown_connections 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5646cee098d0 0x5646cee085e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646ced89710 0x5646ced97d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5646ced88d50 0x5646ced94db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0ce97fa640 0 client.15093 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 --2- 172.21.15.26:0/3493648954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5646ced87b80 0x5646ced94870 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 >> 172.21.15.26:0/3493648954 conn(0x5646ced83070 msgr2=0x5646ced83450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 shutdown_connections 2024-04-23T18:16:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 1 -- 172.21.15.26:0/3493648954 wait complete. 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.849+0000 7f0d0e78e740 20 client.15093 trim_cache size 0 max 16384 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 1, 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24464, 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 86.155717886000005, 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:16:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:16:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:16:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 106.44974491399999 2024-04-23T18:16:35.852 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:16:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 msgr2=0x7f0cec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 0x7f0cec0a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4020000 tx=0x7f0cf401b000 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 msgr2=0x7f0d041981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005ab0 tx=0x7f0cfc003cd0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 shutdown_connections 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0cec0a3c40 0x7f0cec0a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d04105340 0x7f0d04198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d04107c70 0x7f0d0419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 --2- 172.21.15.26:0/3730637124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d04102a10 0x7f0d041981f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 >> 172.21.15.26:0/3730637124 conn(0x7f0d040fc400 msgr2=0x7f0d04103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 shutdown_connections 2024-04-23T18:16:35.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.850+0000 7f0d0ba79640 1 -- 172.21.15.26:0/3730637124 wait complete. 2024-04-23T18:16:35.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:16:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 -- 172.21.15.26:0/1618683195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4101fc0 msgr2=0x7f35c41023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/1618683195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4101fc0 0x7f35c41023e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009900 tx=0x7f35bc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 -- 172.21.15.26:0/1618683195 shutdown_connections 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/1618683195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 0x7f35c4103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/1618683195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4103b70 0x7f35c410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/1618683195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4101fc0 0x7f35c41023e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.963+0000 7f35cb9fe640 1 -- 172.21.15.26:0/1618683195 >> 172.21.15.26:0/1618683195 conn(0x7f35c40fd530 msgr2=0x7f35c40ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- 172.21.15.26:0/1618683195 shutdown_connections 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- 172.21.15.26:0/1618683195 wait complete. 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 Processor -- start 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- start start 2024-04-23T18:16:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4101fc0 0x7f35c4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 0x7f35c4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f35c410db60 con 0x7f35c4103b70 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f35c410d9e0 con 0x7f35c41031b0 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35cb9fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f35c410dce0 con 0x7f35c4101fc0 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35c9f9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35c9f9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38804/0 (socket says 172.21.15.26:38804) 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35c9f9b640 1 -- 172.21.15.26:0/3689336605 learned_addr learned my addr 172.21.15.26:0/3689336605 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.964+0000 7f35c8f99640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 0x7f35c4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c979a640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4101fc0 0x7f35c4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 -- 172.21.15.26:0/3689336605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4101fc0 msgr2=0x7f35c4075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4101fc0 0x7f35c4075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 -- 172.21.15.26:0/3689336605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 msgr2=0x7f35c4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 0x7f35c4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 -- 172.21.15.26:0/3689336605 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c4075110 con 0x7f35c4103b70 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c9f9b640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35b400eaa0 tx=0x7f35b400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c27fc640 1 -- 172.21.15.26:0/3689336605 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4004d20 con 0x7f35c4103b70 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c41a2380 con 0x7f35c4103b70 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c27fc640 1 -- 172.21.15.26:0/3689336605 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b4005020 con 0x7f35c4103b70 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35c27fc640 1 -- 172.21.15.26:0/3689336605 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b40116d0 con 0x7f35c4103b70 2024-04-23T18:16:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.965+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c41a29a0 con 0x7f35c4103b70 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.967+0000 7f35c27fc640 1 -- 172.21.15.26:0/3689336605 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f35b400caf0 con 0x7f35c4103b70 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 Processor -- start 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 -- start start 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004b9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004bb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d10041ec50 con 0x55d1004bb710 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d1003bb830 con 0x55d1004bad50 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d10041ac10 con 0x55d1004b9b80 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35c27fc640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 0x7f35ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38816/0 (socket says 172.21.15.26:38816) 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.968+0000 7f35c8f99640 1 -- 172.21.15.26:0/2949292591 learned_addr learned my addr 172.21.15.26:0/2949292591 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c979a640 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004bb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c9f9b640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 0x7f35ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004b9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c27fc640 1 -- 172.21.15.26:0/3689336605 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f35b40d07c0 con 0x7f35c4103b70 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 -- 172.21.15.26:0/2949292591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 msgr2=0x55d1004b9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004b9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 -- 172.21.15.26:0/2949292591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 msgr2=0x55d1004bb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004bb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 -- 172.21.15.26:0/2949292591 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1001312d0 con 0x55d1004bb710 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c8f99640 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009f30 tx=0x7f35b80095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9cb22a9e7ae4e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35c9f9b640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 0x7f35ac0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f35b401c000 tx=0x7f35b4065440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35ab7fe640 1 -- 172.21.15.26:0/2949292591 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8017610 con 0x55d1004bb710 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35ab7fe640 1 -- 172.21.15.26:0/2949292591 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b801c070 con 0x55d1004bb710 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.969+0000 7f35ab7fe640 1 -- 172.21.15.26:0/2949292591 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8015a10 con 0x55d1004bb710 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- 172.21.15.26:0/2949292591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 msgr2=0x55d1004c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009f30 tx=0x7f35b80095a0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- 172.21.15.26:0/2949292591 shutdown_connections 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004bb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004c1fa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/2949292591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004b9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- 172.21.15.26:0/2949292591 >> 172.21.15.26:0/2949292591 conn(0x55d1004b5070 msgr2=0x55d1004b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- 172.21.15.26:0/2949292591 shutdown_connections 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- 172.21.15.26:0/2949292591 wait complete. 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 -- start start 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.970+0000 7f35ce7ed740 1 Processor -- start 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d1004c2720 con 0x55d1004bb710 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d1003b9470 con 0x55d1004bad50 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d1004c28a0 con 0x55d1004b9b80 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35ce7ed740 10 client.0 _ll_register_callbacks cb 0x55d1003def30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38818/0 (socket says 172.21.15.26:38818) 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 -- 172.21.15.26:0/339991355 learned_addr learned my addr 172.21.15.26:0/339991355 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c979a640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 -- 172.21.15.26:0/339991355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 msgr2=0x55d1004c3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 -- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 msgr2=0x55d1004c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 -- 172.21.15.26:0/339991355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1001fb6b0 con 0x55d1004bb710 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35c8f99640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f35b8006ef0 tx=0x7f35b8010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac4d5ed7661d00b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.971+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8015e50 con 0x55d1004bb710 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f35b8017410 con 0x55d1004bb710 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35b801c070 con 0x55d1004bb710 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35b8017610 con 0x55d1004bb710 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 10 client.15105 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d1004ce8a0 con 0x55d1004bb710 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b801c070 con 0x55d1004bb710 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f35b8032770 con 0x55d1004bb710 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 10 client.15105 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 20 client.15105 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d1004d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 4 client.15105 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d10042c670 con 0x55d1004d0af0 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.972+0000 7f35c979a640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d1004d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.973+0000 7f35c979a640 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d1004d1f60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a99fb8bb43b5d23c server_cookie=4b102f3e7adbf729 in_seq=0 out_seq=0 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.973+0000 7f35a8ff9640 10 client.15105 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35a8ff9640 1 -- 172.21.15.26:0/339991355 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d10042c670 con 0x55d1004d0af0 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35a8ff9640 10 client.15105 handle_command_reply: tid=0 2024-04-23T18:16:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 client.15105 shutdown 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 msgr2=0x55d1004cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f35b8006ef0 tx=0x7f35b8010f90 comp rx=0 tx=0).stop 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 shutdown_connections 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d1004d1f60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bad50 0x55d1004c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bb710 0x55d1004cb530 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35a8ff9640 0 client.15105 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/339991355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 >> 172.21.15.26:0/339991355 conn(0x55d1004b5070 msgr2=0x55d1004b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 shutdown_connections 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 1 -- 172.21.15.26:0/339991355 wait complete. 2024-04-23T18:16:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.974+0000 7f35ce7ed740 20 client.15105 trim_cache size 0 max 16384 2024-04-23T18:16:35.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 Processor -- start 2024-04-23T18:16:35.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 -- start start 2024-04-23T18:16:35.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004c4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d02a0 con 0x55d1004bad50 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d0970 con 0x55d1004bb710 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d1040 con 0x55d1004b9b80 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.992+0000 7f35c979a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38830/0 (socket says 172.21.15.26:38830) 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 -- 172.21.15.26:0/1203179925 learned_addr learned my addr 172.21.15.26:0/1203179925 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 -- 172.21.15.26:0/1203179925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 msgr2=0x55d1004c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 -- 172.21.15.26:0/1203179925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 msgr2=0x55d1004c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 -- 172.21.15.26:0/1203179925 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1004b9560 con 0x55d1004bad50 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35c979a640 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f35bc005ab0 tx=0x7f35bc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b9010a93035687 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35aaffd640 1 -- 172.21.15.26:0/1203179925 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc035040 con 0x55d1004bad50 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35aaffd640 1 -- 172.21.15.26:0/1203179925 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35bc049070 con 0x55d1004bad50 2024-04-23T18:16:35.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.993+0000 7f35aaffd640 1 -- 172.21.15.26:0/1203179925 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0052a0 con 0x55d1004bad50 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- 172.21.15.26:0/1203179925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 msgr2=0x55d1004cb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f35bc005ab0 tx=0x7f35bc005a20 comp rx=0 tx=0).stop 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- 172.21.15.26:0/1203179925 shutdown_connections 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004bad50 0x55d1004cb450 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/1203179925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004b9b80 0x55d1004c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- 172.21.15.26:0/1203179925 >> 172.21.15.26:0/1203179925 conn(0x55d1004b5070 msgr2=0x55d1004be7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- 172.21.15.26:0/1203179925 shutdown_connections 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- 172.21.15.26:0/1203179925 wait complete. 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- start start 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 Processor -- start 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004bad50 0x55d1004c7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d1de0 con 0x55d1004b9b80 2024-04-23T18:16:35.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d1710 con 0x55d1004bb710 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d1004d24b0 con 0x55d1004bad50 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35ce7ed740 10 client.0 _ll_register_callbacks cb 0x7f35bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.994+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38840/0 (socket says 172.21.15.26:38840) 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 -- 172.21.15.26:0/761474043 learned_addr learned my addr 172.21.15.26:0/761474043 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 -- 172.21.15.26:0/761474043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004bad50 msgr2=0x55d1004c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c979a640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004bad50 0x55d1004c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004bad50 0x55d1004c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 -- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 msgr2=0x55d1004ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 -- 172.21.15.26:0/761474043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1004ca7a0 con 0x55d1004b9b80 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35c8f99640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f35b8021420 tx=0x7f35b80069d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818d6c64515ad156 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:16:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8037030 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f35b80173e0 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35bc049070 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35b8015520 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35ce7ed740 10 client.15111 fetch_fsmap learned FSMap version 21 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d100536be0 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.995+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35bc049070 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f35b8032490 con 0x55d1004b9b80 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 10 client.15111 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 20 client.15111 populate_metadata read hostname 'smithi026' 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d100537c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 4 client.15111 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "status"}) -- 0x55d1004264e0 con 0x55d1004d0af0 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35c979a640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d100537c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35a8ff9640 10 client.15111 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.996+0000 7f35c979a640 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d100537c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=323efef6c2aecf3d server_cookie=f04abd96b4a2999e in_seq=0 out_seq=0 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35a8ff9640 1 -- 172.21.15.26:0/761474043 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d1004264e0 con 0x55d1004d0af0 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35a8ff9640 10 client.15111 handle_command_reply: tid=0 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 client.15111 shutdown 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 msgr2=0x55d1004c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f35b8021420 tx=0x7f35b80069d0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 shutdown_connections 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d1004d0af0 0x55d100537c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d1004bb710 0x55d1004ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35a8ff9640 0 client.15111 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1004bad50 0x55d1004c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 --2- 172.21.15.26:0/761474043 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d1004b9b80 0x55d1004c6c50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 >> 172.21.15.26:0/761474043 conn(0x55d1004b5070 msgr2=0x55d1004c3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 shutdown_connections 2024-04-23T18:16:35.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 1 -- 172.21.15.26:0/761474043 wait complete. 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35ce7ed740 20 client.15111 trim_cache size 0 max 16384 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 0, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24503, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 89.322980903000001, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 95.336279789000002 2024-04-23T18:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:16:35.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 msgr2=0x7f35ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 0x7f35ac0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f35b401c000 tx=0x7f35b4065440 comp rx=0 tx=0).stop 2024-04-23T18:16:35.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 msgr2=0x7f35c4074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.997+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35b400eaa0 tx=0x7f35b400eea0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 shutdown_connections 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f35ac0a3c40 0x7f35ac0a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35c41031b0 0x7f35c4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f35c4103b70 0x7f35c4074880 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 --2- 172.21.15.26:0/3689336605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35c4101fc0 0x7f35c4075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 >> 172.21.15.26:0/3689336605 conn(0x7f35c40fd530 msgr2=0x7f35c4106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 shutdown_connections 2024-04-23T18:16:35.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:35.998+0000 7f35cb9fe640 1 -- 172.21.15.26:0/3689336605 wait complete. 2024-04-23T18:16:36.002 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T18:16:36.002 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:16:36.013 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:16:36.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-23T18:16:36.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:36 smithi026 ceph-mon[29364]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:36.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:36 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1949343210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:36 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/127569420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.260 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:36.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:36 smithi110 ceph-mon[32585]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:36.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:36 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1949343210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:36 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/127569420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:36 smithi131 ceph-mon[33041]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-23T18:16:36.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:36 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1949343210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:36.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:36 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/127569420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 -- 172.21.15.26:0/3785577320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4108810 msgr2=0x7fabe410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 --2- 172.21.15.26:0/3785577320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4108810 0x7fabe410ac20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009990 tx=0x7fabdc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 -- 172.21.15.26:0/3785577320 shutdown_connections 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 --2- 172.21.15.26:0/3785577320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4108810 0x7fabe410ac20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 --2- 172.21.15.26:0/3785577320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe41035b0 0x7fabe41059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 --2- 172.21.15.26:0/3785577320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe4105ee0 0x7fabe41082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 -- 172.21.15.26:0/3785577320 >> 172.21.15.26:0/3785577320 conn(0x7fabe40fcf60 msgr2=0x7fabe40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 -- 172.21.15.26:0/3785577320 shutdown_connections 2024-04-23T18:16:37.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 -- 172.21.15.26:0/3785577320 wait complete. 2024-04-23T18:16:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.993+0000 7fabe9e3c700 1 Processor -- start 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 -- start start 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe41035b0 0x7fabe4102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe4108810 0x7fabe4101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410e550 con 0x7fabe4105ee0 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410e3d0 con 0x7fabe4108810 2024-04-23T18:16:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe9e3c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410e6d0 con 0x7fabe41035b0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45138/0 (socket says 172.21.15.26:45138) 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 -- 172.21.15.26:0/2804601694 learned_addr learned my addr 172.21.15.26:0/2804601694 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe37fe700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe41035b0 0x7fabe4102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe3fff700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe4108810 0x7fabe4101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 -- 172.21.15.26:0/2804601694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe41035b0 msgr2=0x7fabe4102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe41035b0 0x7fabe4102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 -- 172.21.15.26:0/2804601694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe4108810 msgr2=0x7fabe4101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe4108810 0x7fabe4101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.994+0000 7fabe2ffd700 1 -- 172.21.15.26:0/2804601694 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe4102350 con 0x7fabe4105ee0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe2ffd700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00d980 tx=0x7fabcc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc017070 con 0x7fabe4105ee0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe9e3c700 1 -- 172.21.15.26:0/2804601694 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe41a2d30 con 0x7fabe4105ee0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabcc005020 con 0x7fabe4105ee0 2024-04-23T18:16:37.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00fa10 con 0x7fabe4105ee0 2024-04-23T18:16:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.995+0000 7fabe9e3c700 1 -- 172.21.15.26:0/2804601694 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe41a3190 con 0x7fabe4105ee0 2024-04-23T18:16:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.996+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc8005330 con 0x7fabe4105ee0 2024-04-23T18:16:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.997+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fabcc00fbb0 con 0x7fabe4105ee0 2024-04-23T18:16:37.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.998+0000 7fabe0ff9700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 0x7fabd00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.998+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fabcc0d03c0 con 0x7fabe4105ee0 2024-04-23T18:16:37.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.998+0000 7fabe37fe700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 0x7fabd00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:37.999+0000 7fabe37fe700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 0x7fabd00a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009550 tx=0x7fabd40091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:37.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.003+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabcc020020 con 0x7fabe4105ee0 2024-04-23T18:16:38.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.232+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fabc8000bc0 con 0x7fabd00a3cd0 2024-04-23T18:16:38.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.233+0000 7fabe0ff9700 1 -- 172.21.15.26:0/2804601694 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fabc8000bc0 con 0x7fabd00a3cd0 2024-04-23T18:16:38.226 INFO:teuthology.orchestra.run.smithi026.stdout:Backend: cephadm 2024-04-23T18:16:38.226 INFO:teuthology.orchestra.run.smithi026.stdout:Available: Yes 2024-04-23T18:16:38.227 INFO:teuthology.orchestra.run.smithi026.stdout:Paused: No 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 msgr2=0x7fabd00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 0x7fabd00a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009550 tx=0x7fabd40091b0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 msgr2=0x7fabe4101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00d980 tx=0x7fabcc00dc90 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 shutdown_connections 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fabd00a3cd0 0x7fabd00a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fabe4108810 0x7fabe4101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe41035b0 0x7fabe4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 --2- 172.21.15.26:0/2804601694 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fabe4105ee0 0x7fabe4101560 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 >> 172.21.15.26:0/2804601694 conn(0x7fabe40fcf60 msgr2=0x7fabe4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 shutdown_connections 2024-04-23T18:16:38.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:38.235+0000 7fabda7fc700 1 -- 172.21.15.26:0/2804601694 wait complete. 2024-04-23T18:16:38.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:38 smithi026 ceph-mon[29364]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:38.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:38 smithi110 ceph-mon[32585]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:38.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:38 smithi131 ceph-mon[33041]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:38.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-23T18:16:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:39.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:39 smithi026 ceph-mon[29364]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:39.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:39 smithi110 ceph-mon[32585]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:39.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:39 smithi131 ceph-mon[33041]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.244 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:40 smithi026 ceph-mon[29364]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:40.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:40 smithi110 ceph-mon[32585]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:40.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:40 smithi131 ceph-mon[33041]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:16:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 -- 172.21.15.26:0/3911574351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f8103540 msgr2=0x7f32f8109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3911574351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f8103540 0x7f32f8109df0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f32f000b0d0 tx=0x7f32f000b3e0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 -- 172.21.15.26:0/3911574351 shutdown_connections 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3911574351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f8103540 0x7f32f8109df0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3911574351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f81019a0 0x7f32f8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3911574351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 0x7f32f8103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.787+0000 7f32fcd79700 1 -- 172.21.15.26:0/3911574351 >> 172.21.15.26:0/3911574351 conn(0x7f32f80fcf40 msgr2=0x7f32f80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.788+0000 7f32fcd79700 1 -- 172.21.15.26:0/3911574351 shutdown_connections 2024-04-23T18:16:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.788+0000 7f32fcd79700 1 -- 172.21.15.26:0/3911574351 wait complete. 2024-04-23T18:16:40.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.788+0000 7f32fcd79700 1 Processor -- start 2024-04-23T18:16:40.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 -- start start 2024-04-23T18:16:40.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 0x7f32f81990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 0x7f32f81a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d470 con 0x7f32f81019a0 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d2f0 con 0x7f32f8103540 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32fcd79700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d5f0 con 0x7f32f8102ba0 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45158/0 (socket says 172.21.15.26:45158) 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 -- 172.21.15.26:0/3099040448 learned_addr learned my addr 172.21.15.26:0/3099040448 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f5d9b700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 0x7f32f81990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f6d9d700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 0x7f32f81a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 -- 172.21.15.26:0/3099040448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 msgr2=0x7f32f81990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 0x7f32f81990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 -- 172.21.15.26:0/3099040448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 msgr2=0x7f32f81a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 0x7f32f81a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f659c700 1 -- 172.21.15.26:0/3099040448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f81a0810 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.789+0000 7f32f6d9d700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 0x7f32f81a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32f659c700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00c9c0 tx=0x7f32ec00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec007ac0 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32ec007c60 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f81a0b00 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec018660 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.790+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f81a1040 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.791+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32f8069730 con 0x7f32f81019a0 2024-04-23T18:16:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.793+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f32ec00f450 con 0x7f32f81019a0 2024-04-23T18:16:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.793+0000 7f32e77fe700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 0x7f32e00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.794+0000 7f32f5d9b700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 0x7f32e00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.794+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f32ec0d0730 con 0x7f32f81019a0 2024-04-23T18:16:40.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.794+0000 7f32f5d9b700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 0x7f32e00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f32e8005950 tx=0x7f32e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:40.797+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32ec014030 con 0x7f32f81019a0 2024-04-23T18:16:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.042+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f32ec00f740 con 0x7f32f81019a0 2024-04-23T18:16:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.049+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f32f80ff0f0 con 0x7f32e00a3cd0 2024-04-23T18:16:41.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.064+0000 7f32e77fe700 1 -- 172.21.15.26:0/3099040448 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f32f80ff0f0 con 0x7f32e00a3cd0 2024-04-23T18:16:41.059 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T18:16:41.059 INFO:teuthology.orchestra.run.smithi026.stdout:mds.a smithi026 running (103s) 18s ago 102s 13.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 eaedc3689afc 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.b smithi110 running (100s) 34s ago 100s 28.5M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2e93d24b46a6 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.c smithi131 running (114s) 34s ago 114s 17.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cf63d63b7abe 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.d smithi026 running (2m) 18s ago 2m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e73333c29717 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.e smithi110 running (2m) 34s ago 119s 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 76fdb3eaaa20 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.f smithi131 running (117s) 34s ago 117s 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e706f4ff5f2b 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.g smithi026 running (106s) 18s ago 105s 15.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 91126b77f56b 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.h smithi110 running (112s) 34s ago 111s 17.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 709e020b41de 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.i smithi131 running (109s) 34s ago 108s 17.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 a8c621aae573 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.j smithi026 running (2m) 18s ago 2m 13.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 3e4e9a7b1974 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.k smithi110 running (2m) 34s ago 2m 15.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 68c56aee4ffc 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mds.l smithi131 running (2m) 34s ago 2m 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 1d1ff2f5866c 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.x smithi026 *:9283,8765 running (9m) 18s ago 9m 514M - 19.0.0-3156-gd69f13d8 da4b004c09e8 0403e4f3d070 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.y smithi110 *:8443,8765 running (7m) 34s ago 7m 454M - 19.0.0-3156-gd69f13d8 da4b004c09e8 dffa7376cdd3 2024-04-23T18:16:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.z smithi131 *:8443,8765 running (7m) 34s ago 7m 452M - 19.0.0-3156-gd69f13d8 da4b004c09e8 4fc2e7ecaeb9 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a smithi026 running (9m) 18s ago 9m 52.9M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 d69906676dfc 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b smithi110 running (7m) 34s ago 7m 37.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 cb6ebff28cab 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c smithi131 running (7m) 34s ago 7m 40.6M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 d4b4b575b5e6 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (7m) 18s ago 7m 58.2M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 3c1b9ad246c4 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi110 running (6m) 34s ago 6m 48.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 32861525d380 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi131 running (6m) 34s ago 6m 57.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 214b83086729 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi026 running (5m) 18s ago 5m 46.1M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 9af49edcd13c 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi110 running (5m) 34s ago 5m 51.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 75f5d13b7690 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi131 running (5m) 34s ago 5m 55.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 18ac30b513fd 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 smithi026 running (4m) 18s ago 4m 44.0M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 017740441c30 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 smithi110 running (4m) 34s ago 4m 55.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 e0ba6e983356 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8 smithi131 running (3m) 34s ago 3m 46.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 1d57784ee194 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9 smithi026 running (3m) 18s ago 3m 50.2M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 7612ab22c2b8 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10 smithi110 running (2m) 34s ago 2m 46.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 eab00604fb0e 2024-04-23T18:16:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 smithi131 running (2m) 34s ago 2m 52.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 a866c9c0eb06 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 msgr2=0x7f32e00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 0x7f32e00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f32e8005950 tx=0x7f32e800b410 comp rx=0 tx=0).stop 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 msgr2=0x7f32f8198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00c9c0 tx=0x7f32ec00ccd0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 shutdown_connections 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f32e00a3cd0 0x7f32e00a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32f8103540 0x7f32f81a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32f81019a0 0x7f32f8198bb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 --2- 172.21.15.26:0/3099040448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32f8102ba0 0x7f32f81990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 >> 172.21.15.26:0/3099040448 conn(0x7f32f80fcf40 msgr2=0x7f32f80fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 shutdown_connections 2024-04-23T18:16:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:41.067+0000 7f32fcd79700 1 -- 172.21.15.26:0/3099040448 wait complete. 2024-04-23T18:16:41.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-23T18:16:41.993 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:42.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:42 smithi026 ceph-mon[29364]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:42.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:42 smithi026 ceph-mon[29364]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-23T18:16:42.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:42 smithi026 ceph-mon[29364]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:42.303 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:42 smithi131 ceph-mon[33041]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:42.303 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:42 smithi131 ceph-mon[33041]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-23T18:16:42.303 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:42 smithi131 ceph-mon[33041]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:42.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:42 smithi110 ceph-mon[32585]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:16:42.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:42 smithi110 ceph-mon[32585]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-23T18:16:42.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:42 smithi110 ceph-mon[32585]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.714+0000 7f11a8c17700 1 -- 172.21.15.26:0/3204192029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4108810 msgr2=0x7f11a410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.714+0000 7f11a8c17700 1 --2- 172.21.15.26:0/3204192029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4108810 0x7f11a410ac20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f119c00b0a0 tx=0x7f119c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 -- 172.21.15.26:0/3204192029 shutdown_connections 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 --2- 172.21.15.26:0/3204192029 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4108810 0x7f11a410ac20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 --2- 172.21.15.26:0/3204192029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 0x7f11a41059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 --2- 172.21.15.26:0/3204192029 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4105ee0 0x7f11a41082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 -- 172.21.15.26:0/3204192029 >> 172.21.15.26:0/3204192029 conn(0x7f11a40fcf60 msgr2=0x7f11a40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 -- 172.21.15.26:0/3204192029 shutdown_connections 2024-04-23T18:16:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.715+0000 7f11a8c17700 1 -- 172.21.15.26:0/3204192029 wait complete. 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 Processor -- start 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 -- start start 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 0x7f11a4102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4108810 0x7f11a4101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f11a410e550 con 0x7f11a4105ee0 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f11a410e3d0 con 0x7f11a41035b0 2024-04-23T18:16:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a8c17700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f11a410e6d0 con 0x7f11a4108810 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a1d9b700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45170/0 (socket says 172.21.15.26:45170) 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a1d9b700 1 -- 172.21.15.26:0/339442005 learned_addr learned my addr 172.21.15.26:0/339442005 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a2d9d700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4108810 0x7f11a4101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a259c700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 0x7f11a4102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a1d9b700 1 -- 172.21.15.26:0/339442005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4108810 msgr2=0x7f11a4101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.716+0000 7f11a1d9b700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4108810 0x7f11a4101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a1d9b700 1 -- 172.21.15.26:0/339442005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 msgr2=0x7f11a4102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a1d9b700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 0x7f11a4102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a1d9b700 1 -- 172.21.15.26:0/339442005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a4102350 con 0x7f11a4105ee0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a1d9b700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f119800d980 tx=0x7f119800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198017070 con 0x7f11a4105ee0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1198005020 con 0x7f11a4105ee0 2024-04-23T18:16:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a41a2ca0 con 0x7f11a4105ee0 2024-04-23T18:16:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119800fa10 con 0x7f11a4105ee0 2024-04-23T18:16:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.717+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a41a3160 con 0x7f11a4105ee0 2024-04-23T18:16:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.718+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a4069730 con 0x7f11a4105ee0 2024-04-23T18:16:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.719+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1198004a30 con 0x7f11a4105ee0 2024-04-23T18:16:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.720+0000 7f11937fe700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 0x7f118c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.720+0000 7f11a259c700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 0x7f118c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.720+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f11980d0270 con 0x7f11a4105ee0 2024-04-23T18:16:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.720+0000 7f11a259c700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 0x7f118c0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f119400ac50 tx=0x7f119400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.725+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1198020020 con 0x7f11a4105ee0 2024-04-23T18:16:43.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.945+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f11a4062e80 con 0x7f118c0a3c40 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.951+0000 7f11937fe700 1 -- 172.21.15.26:0/339442005 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f11a4062e80 con 0x7f118c0a3c40 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stdout:mds.all 12/12 37s ago 2m smithi026=a;smithi026=d;smithi026=g;smithi026=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stdout:mgr 3/3 37s ago 7m smithi026=x;smithi110=y;smithi131=z;count:3 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stdout:mon 3/3 37s ago 7m smithi026:172.21.15.26=a;smithi110:172.21.15.110=b;smithi131:172.21.15.131=c;count:3 2024-04-23T18:16:43.945 INFO:teuthology.orchestra.run.smithi026.stdout:osd 12 37s ago - 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 msgr2=0x7f118c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 0x7f118c0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f119400ac50 tx=0x7f119400a380 comp rx=0 tx=0).stop 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 msgr2=0x7f11a4101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f119800d980 tx=0x7f119800dc90 comp rx=0 tx=0).stop 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 shutdown_connections 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f118c0a3c40 0x7f118c0a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11a4108810 0x7f11a4101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a41035b0 0x7f11a4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 --2- 172.21.15.26:0/339442005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f11a4105ee0 0x7f11a4101560 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 >> 172.21.15.26:0/339442005 conn(0x7f11a40fcf60 msgr2=0x7f11a4103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 shutdown_connections 2024-04-23T18:16:43.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:43.953+0000 7f11a8c17700 1 -- 172.21.15.26:0/339442005 wait complete. 2024-04-23T18:16:44.315 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:44 smithi026 ceph-mon[29364]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:16:44.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:44 smithi110 ceph-mon[32585]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:16:44.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:44 smithi131 ceph-mon[33041]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T18:16:44.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-23T18:16:44.948 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:45.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:45 smithi026 ceph-mon[29364]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:45.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:45 smithi110 ceph-mon[32585]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:45.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:45 smithi131 ceph-mon[33041]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:46.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:46 smithi026 ceph-mon[29364]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:16:46.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:46 smithi110 ceph-mon[32585]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:16:46.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:46 smithi131 ceph-mon[33041]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.701+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/2756114423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 msgr2=0x7f2e9810af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.701+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/2756114423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e9810af10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000b0d0 tx=0x7f2e9000b3e0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/2756114423 shutdown_connections 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/2756114423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e9810af10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/2756114423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98102ac0 0x7f2e98102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/2756114423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98103cc0 0x7f2e98104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/2756114423 >> 172.21.15.26:0/2756114423 conn(0x7f2e980fe060 msgr2=0x7f2e981004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/2756114423 shutdown_connections 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/2756114423 wait complete. 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 Processor -- start 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.702+0000 7f2e9d5b4700 1 -- start start 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98102ac0 0x7f2e98198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98103cc0 0x7f2e98198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810e540 con 0x7f2e98104660 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810e3c0 con 0x7f2e98103cc0 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810e6c0 con 0x7f2e98102ac0 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:46.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45188/0 (socket says 172.21.15.26:45188) 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 -- 172.21.15.26:0/3237563415 learned_addr learned my addr 172.21.15.26:0/3237563415 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e96ffd700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98102ac0 0x7f2e98198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e967fc700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98103cc0 0x7f2e98198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 -- 172.21.15.26:0/3237563415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98102ac0 msgr2=0x7f2e98198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98102ac0 0x7f2e98198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 -- 172.21.15.26:0/3237563415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98103cc0 msgr2=0x7f2e98198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98103cc0 0x7f2e98198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 -- 172.21.15.26:0/3237563415 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e981a0700 con 0x7f2e98104660 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e977fe700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000d010 tx=0x7f2e90004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9001f030 con 0x7f2e98104660 2024-04-23T18:16:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e981a0990 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.704+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981a0e40 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.703+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e90003a00 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.704+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90004740 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.704+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e98069730 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.706+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2e90007a00 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.706+0000 7f2e7ffff700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 0x7f2e800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.706+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e9000f020 con 0x7f2e98104660 2024-04-23T18:16:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.707+0000 7f2e96ffd700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 0x7f2e800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.707+0000 7f2e96ffd700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 0x7f2e800a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2e98103b20 tx=0x7f2e8800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.710+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e90029020 con 0x7f2e98104660 2024-04-23T18:16:46.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.925+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f2e98062e80 con 0x7f2e800a3c40 2024-04-23T18:16:46.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.926+0000 7f2e7ffff700 1 -- 172.21.15.26:0/3237563415 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f2e98062e80 con 0x7f2e800a3c40 2024-04-23T18:16:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:HOST ADDR LABELS STATUS 2024-04-23T18:16:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 172.21.15.26 2024-04-23T18:16:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:smithi110 172.21.15.110 2024-04-23T18:16:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:smithi131 172.21.15.131 2024-04-23T18:16:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:3 hosts in cluster 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 msgr2=0x7f2e800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 0x7f2e800a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2e98103b20 tx=0x7f2e8800a3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 msgr2=0x7f2e981a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000d010 tx=0x7f2e90004e40 comp rx=0 tx=0).stop 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 shutdown_connections 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2e800a3c40 0x7f2e800a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2e98104660 0x7f2e981a0060 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e98102ac0 0x7f2e98198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 --2- 172.21.15.26:0/3237563415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e98103cc0 0x7f2e98198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 >> 172.21.15.26:0/3237563415 conn(0x7f2e980fe060 msgr2=0x7f2e98106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:46.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 shutdown_connections 2024-04-23T18:16:46.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:46.929+0000 7f2e9d5b4700 1 -- 172.21.15.26:0/3237563415 wait complete. 2024-04-23T18:16:47.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-23T18:16:47.980 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:48.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:48 smithi026 ceph-mon[29364]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:48.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:48 smithi026 ceph-mon[29364]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:48.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:48 smithi110 ceph-mon[32585]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:48.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:48 smithi110 ceph-mon[32585]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:48.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:48 smithi131 ceph-mon[33041]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:48.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:48 smithi131 ceph-mon[33041]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:49.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 -- 172.21.15.26:0/3350631351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42701087d0 msgr2=0x7f427010ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 --2- 172.21.15.26:0/3350631351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42701087d0 0x7f427010ac40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f426800b0a0 tx=0x7f426800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 -- 172.21.15.26:0/3350631351 shutdown_connections 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 --2- 172.21.15.26:0/3350631351 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42701087d0 0x7f427010ac40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 --2- 172.21.15.26:0/3350631351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 0x7f4270105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 --2- 172.21.15.26:0/3350631351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4270105ea0 0x7f4270108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 -- 172.21.15.26:0/3350631351 >> 172.21.15.26:0/3350631351 conn(0x7f42700fcfa0 msgr2=0x7f42700ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 -- 172.21.15.26:0/3350631351 shutdown_connections 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.701+0000 7f427752d700 1 -- 172.21.15.26:0/3350631351 wait complete. 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 Processor -- start 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 -- start start 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 0x7f4270196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42701087d0 0x7f42701a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:49.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f427010e560 con 0x7f4270105ea0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f427010e3e0 con 0x7f42701087d0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f427752d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f427010e6e0 con 0x7f4270103570 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60920/0 (socket says 172.21.15.26:60920) 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 -- 172.21.15.26:0/3235641919 learned_addr learned my addr 172.21.15.26:0/3235641919 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f42752c9700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 0x7f4270196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4275aca700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42701087d0 0x7f42701a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 -- 172.21.15.26:0/3235641919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 msgr2=0x7f4270196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 0x7f4270196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 -- 172.21.15.26:0/3235641919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42701087d0 msgr2=0x7f42701a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42701087d0 0x7f42701a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.702+0000 7f4274ac8700 1 -- 172.21.15.26:0/3235641919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42701a6e00 con 0x7f4270105ea0 2024-04-23T18:16:49.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f4274ac8700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f426000d950 tx=0x7f426000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260016070 con 0x7f4270105ea0 2024-04-23T18:16:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42701a70f0 con 0x7f4270105ea0 2024-04-23T18:16:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f426000fd40 con 0x7f4270105ea0 2024-04-23T18:16:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000e3b0 con 0x7f4270105ea0 2024-04-23T18:16:49.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.703+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42701a7650 con 0x7f4270105ea0 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.704+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4270069730 con 0x7f4270105ea0 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.705+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4260019070 con 0x7f4270105ea0 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.705+0000 7f42667fc700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 0x7f425c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.706+0000 7f42752c9700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 0x7f425c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.706+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f42600cfea0 con 0x7f4270105ea0 2024-04-23T18:16:49.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.706+0000 7f42752c9700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 0x7f425c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f426c009c60 tx=0x7f426c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:49.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.710+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f426009c890 con 0x7f4270105ea0 2024-04-23T18:16:49.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.926+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4270062e80 con 0x7f425c0a3c40 2024-04-23T18:16:49.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.928+0000 7f42667fc700 1 -- 172.21.15.26:0/3235641919 <== mgr.14152 v2:172.21.15.26:6800/270207440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4270062e80 con 0x7f425c0a3c40 2024-04-23T18:16:49.922 INFO:teuthology.orchestra.run.smithi026.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-23T18:16:49.922 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760020480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-23T18:16:49.922 INFO:teuthology.orchestra.run.smithi026.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:16:49.922 INFO:teuthology.orchestra.run.smithi026.stdout:smithi131 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300N4400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 msgr2=0x7f425c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 0x7f425c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f426c009c60 tx=0x7f426c009260 comp rx=0 tx=0).stop 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 msgr2=0x7f4270196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f426000d950 tx=0x7f426000dc60 comp rx=0 tx=0).stop 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 shutdown_connections 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f425c0a3c40 0x7f425c0a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42701087d0 0x7f42701a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4270103570 0x7f4270196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 --2- 172.21.15.26:0/3235641919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4270105ea0 0x7f4270196eb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 >> 172.21.15.26:0/3235641919 conn(0x7f42700fcfa0 msgr2=0x7f4270102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 shutdown_connections 2024-04-23T18:16:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:49.930+0000 7f427752d700 1 -- 172.21.15.26:0/3235641919 wait complete. 2024-04-23T18:16:50.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:50 smithi026 ceph-mon[29364]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:50.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:50 smithi110 ceph-mon[32585]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:50.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:50 smithi131 ceph-mon[33041]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:50.801 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:16:50.810 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:16:50.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T18:16:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:51.324 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:51 smithi026 ceph-mon[29364]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:51.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:51 smithi110 ceph-mon[32585]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:51.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:51 smithi131 ceph-mon[33041]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:52.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:52 smithi026 ceph-mon[29364]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:16:52.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:52 smithi110 ceph-mon[32585]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:16:52.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:52 smithi131 ceph-mon[33041]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:16:52.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 -- 172.21.15.26:0/467122174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c941019a0 msgr2=0x7f2c94101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:52.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/467122174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c941019a0 0x7f2c94101dc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2c840099c0 tx=0x7f2c84009cd0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 -- 172.21.15.26:0/467122174 shutdown_connections 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/467122174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 0x7f2c94109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/467122174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c941019a0 0x7f2c94101dc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/467122174 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c94102ba0 0x7f2c94103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 -- 172.21.15.26:0/467122174 >> 172.21.15.26:0/467122174 conn(0x7f2c940fcf40 msgr2=0x7f2c940ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 -- 172.21.15.26:0/467122174 shutdown_connections 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.838+0000 7f2c9bd40700 1 -- 172.21.15.26:0/467122174 wait complete. 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 Processor -- start 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 -- start start 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c941019a0 0x7f2c94198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 0x7f2c941a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:52.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9410d490 con 0x7f2c94102ba0 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9410d310 con 0x7f2c94103540 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c9bd40700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9410d610 con 0x7f2c941019a0 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.839+0000 7f2c992db700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60938/0 (socket says 172.21.15.26:60938) 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 -- 172.21.15.26:0/2076284237 learned_addr learned my addr 172.21.15.26:0/2076284237 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c99adc700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c941019a0 0x7f2c94198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c9a2dd700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 0x7f2c941a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 -- 172.21.15.26:0/2076284237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c941019a0 msgr2=0x7f2c94198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c941019a0 0x7f2c94198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 -- 172.21.15.26:0/2076284237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 msgr2=0x7f2c941a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 0x7f2c941a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:52.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 -- 172.21.15.26:0/2076284237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c941a0980 con 0x7f2c94102ba0 2024-04-23T18:16:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c992db700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b530 tx=0x7f2c9000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90015020 con 0x7f2c94102ba0 2024-04-23T18:16:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c94103e20 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.840+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c90011ce0 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.841+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90010dd0 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.841+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c94104330 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.842+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c94069730 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.843+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2c90018020 con 0x7f2c94102ba0 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.843+0000 7f2c8affd700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 0x7f2c800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.843+0000 7f2c99adc700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 0x7f2c800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.843+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2c900cfb10 con 0x7f2c94102ba0 2024-04-23T18:16:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.844+0000 7f2c99adc700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 0x7f2c800a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006010 tx=0x7f2c84003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:52.848+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c9001f020 con 0x7f2c94102ba0 2024-04-23T18:16:53.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.104+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2c94104c90 con 0x7f2c94102ba0 2024-04-23T18:16:53.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.104+0000 7f2c8affd700 1 -- 172.21.15.26:0/2076284237 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3058 (secure 0 0 0) 0x7f2c9009c500 con 0x7f2c94102ba0 2024-04-23T18:16:53.098 INFO:teuthology.orchestra.run.smithi026.stdout:e21 2024-04-23T18:16:53.098 INFO:teuthology.orchestra.run.smithi026.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:16:53.098 INFO:teuthology.orchestra.run.smithi026.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:legacy client fscid: 1 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:Filesystem 'cephfs' (1) 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:fs_name cephfs 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 21 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-23T18:15:06.048360+0000 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-23T18:15:11.649607+0000 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:tableserver 0 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:root 0 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:session_timeout 300 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:session_autoclose 300 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:max_file_size 1099511627776 2024-04-23T18:16:53.099 INFO:teuthology.orchestra.run.smithi026.stdout:max_xattr_size 65536 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:required_client_features {} 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure 0 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:last_failure_osd_epoch 0 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:max_mds 3 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:in 0,1,2 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:up {0=24503,1=24464,2=24460} 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:failed 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:damaged 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:stopped 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:data_pools [3] 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:metadata_pool 2 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:inline_data disabled 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:balancer 2024-04-23T18:16:53.100 INFO:teuthology.orchestra.run.smithi026.stdout:bal_rank_mask -1 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:standby_count_wanted 0 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:qdb_cluster leader: 24503 members: 24464,24503,24460 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.c{0:24445} state up:standby-replay seq 1 addr [v2:172.21.15.131:6836/2204987461,v1:172.21.15.131:6837/2204987461] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.h{1:24464} state up:active seq 7 addr [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.e{1:24437} state up:standby-replay seq 1 addr [v2:172.21.15.110:6834/3666168856,v1:172.21.15.110:6835/3666168856] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.i{2:24460} state up:active seq 6 addr [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.f{2:24433} state up:standby-replay seq 1 addr [v2:172.21.15.131:6834/2220145154,v1:172.21.15.131:6835/2220145154] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout:Standby daemons: 2024-04-23T18:16:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.j{-1:14565} state up:standby seq 1 addr [v2:172.21.15.26:6834/1467060531,v1:172.21.15.26:6835/1467060531] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.d{-1:14583} state up:standby seq 1 addr [v2:172.21.15.26:6836/1500031392,v1:172.21.15.26:6837/1500031392] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.g{-1:14643} state up:standby seq 1 addr [v2:172.21.15.26:6838/1936694091,v1:172.21.15.26:6839/1936694091] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.a{-1:14655} state up:standby seq 1 addr [v2:172.21.15.26:6840/2731986985,v1:172.21.15.26:6841/2731986985] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.l{-1:24400} state up:standby seq 1 addr [v2:172.21.15.131:6832/2977568812,v1:172.21.15.131:6833/2977568812] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.102 INFO:teuthology.orchestra.run.smithi026.stdout:[mds.k{-1:24407} state up:standby seq 1 addr [v2:172.21.15.110:6832/1312481724,v1:172.21.15.110:6833/1312481724] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 msgr2=0x7f2c800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 0x7f2c800a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006010 tx=0x7f2c84003680 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 msgr2=0x7f2c941992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b530 tx=0x7f2c9000b840 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 shutdown_connections 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f2c800a3c40 0x7f2c800a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c94103540 0x7f2c941a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c941019a0 0x7f2c94198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 --2- 172.21.15.26:0/2076284237 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2c94102ba0 0x7f2c941992c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:53.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 >> 172.21.15.26:0/2076284237 conn(0x7f2c940fcf40 msgr2=0x7f2c9406a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 shutdown_connections 2024-04-23T18:16:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:53.108+0000 7f2c9bd40700 1 -- 172.21.15.26:0/2076284237 wait complete. 2024-04-23T18:16:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:16:53.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-23T18:16:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:54.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:54 smithi026 ceph-mon[29364]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:54.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:54 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2076284237' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:16:54.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:54 smithi110 ceph-mon[32585]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:54.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:54 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2076284237' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:16:54.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:54 smithi131 ceph-mon[33041]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:16:54.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:54 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2076284237' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:16:55.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 -- 172.21.15.26:0/4018539768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc103540 msgr2=0x7f5efc109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 --2- 172.21.15.26:0/4018539768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc103540 0x7f5efc109df0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400b0d0 tx=0x7f5ef400b3e0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 -- 172.21.15.26:0/4018539768 shutdown_connections 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 --2- 172.21.15.26:0/4018539768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc103540 0x7f5efc109df0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 --2- 172.21.15.26:0/4018539768 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc1019a0 0x7f5efc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 --2- 172.21.15.26:0/4018539768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc102ba0 0x7f5efc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 -- 172.21.15.26:0/4018539768 >> 172.21.15.26:0/4018539768 conn(0x7f5efc0fcf40 msgr2=0x7f5efc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:55.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 -- 172.21.15.26:0/4018539768 shutdown_connections 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.756+0000 7f5f010ba700 1 -- 172.21.15.26:0/4018539768 wait complete. 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 Processor -- start 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 -- start start 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc103540 0x7f5efc1044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10d440 con 0x7f5efc1019a0 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10d2c0 con 0x7f5efc103540 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.757+0000 7f5f010ba700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10d5c0 con 0x7f5efc102ba0 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:55.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:46706/0 (socket says 172.21.15.26:46706) 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efad9d700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60948/0 (socket says 172.21.15.26:60948) 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 -- 172.21.15.26:0/3429833875 learned_addr learned my addr 172.21.15.26:0/3429833875 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efb59e700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc103540 0x7f5efc1044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 -- 172.21.15.26:0/3429833875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc103540 msgr2=0x7f5efc1044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc103540 0x7f5efc1044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 -- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 msgr2=0x7f5efc105930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 -- 172.21.15.26:0/3429833875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc104d70 con 0x7f5efc102ba0 2024-04-23T18:16:55.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efad9d700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:55.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.758+0000 7f5efa59c700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000d950 tx=0x7f5ef000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:55.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.759+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0016070 con 0x7f5efc102ba0 2024-04-23T18:16:55.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.759+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5efc1a2de0 con 0x7f5efc102ba0 2024-04-23T18:16:55.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.759+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ef0009df0 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.759+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000f3c0 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.759+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5efc1a3240 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.760+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5efc069730 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.761+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5ef0019070 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.762+0000 7f5ee3fff700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 0x7f5ee40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.762+0000 7f5efad9d700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 0x7f5ee40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.762+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5ef00cfff0 con 0x7f5efc102ba0 2024-04-23T18:16:55.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.762+0000 7f5efad9d700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 0x7f5ee40a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5efc102a00 tx=0x7f5eec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:55.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.766+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ef009c9a0 con 0x7f5efc102ba0 2024-04-23T18:16:55.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.973+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f5efc1a3710 con 0x7f5efc102ba0 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.974+0000 7f5ee3fff700 1 -- 172.21.15.26:0/3429833875 <== mon.1 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5286 (secure 0 0 0) 0x7f5ef009c2a0 con 0x7f5efc102ba0 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 88 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-23T18:07:16.481986+0000 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-04-23T18:16:12.288554+0000 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:crush_version 26 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:full_ratio 0.95 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:backfillfull_ratio 0.9 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:nearfull_ratio 0.85 2024-04-23T18:16:55.968 INFO:teuthology.orchestra.run.smithi026.stdout:require_min_compat_client luminous 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:min_compat_client luminous 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:require_osd_release squid 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:stretch_mode_enabled false 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:max_osd 12 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6802/75846197,v1:172.21.15.26:6803/75846197] [v2:172.21.15.26:6804/75846197,v1:172.21.15.26:6805/75846197] exists,up 6232862c-582d-4b53-96c1-a331e0e5849d 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/1354854884,v1:172.21.15.110:6801/1354854884] [v2:172.21.15.110:6802/1354854884,v1:172.21.15.110:6803/1354854884] exists,up ea35a550-d935-4416-9e66-29ae820b9e4a 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6800/2462004280,v1:172.21.15.131:6801/2462004280] [v2:172.21.15.131:6802/2462004280,v1:172.21.15.131:6803/2462004280] exists,up 09b22982-126d-46bf-9e92-d1792b8c8477 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6810/3619192786,v1:172.21.15.26:6811/3619192786] [v2:172.21.15.26:6812/3619192786,v1:172.21.15.26:6813/3619192786] exists,up 3f222d1e-da51-4d64-8e76-53e764356085 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6808/2989079623,v1:172.21.15.110:6809/2989079623] [v2:172.21.15.110:6810/2989079623,v1:172.21.15.110:6811/2989079623] exists,up 3834c24f-b7b4-41f8-8878-1c54b691ff48 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6808/3617441871,v1:172.21.15.131:6809/3617441871] [v2:172.21.15.131:6810/3617441871,v1:172.21.15.131:6811/3617441871] exists,up 12b632a3-1a0b-4918-9a4d-ccc144053d30 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6818/3473982661,v1:172.21.15.26:6819/3473982661] [v2:172.21.15.26:6820/3473982661,v1:172.21.15.26:6821/3473982661] exists,up 6f5e9d18-f986-4bd6-87b6-ceb84b684bf7 2024-04-23T18:16:55.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6816/3458351312,v1:172.21.15.110:6817/3458351312] [v2:172.21.15.110:6818/3458351312,v1:172.21.15.110:6819/3458351312] exists,up f090001e-2004-4cf8-bcf2-38a16b9618e6 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6816/3490838385,v1:172.21.15.131:6817/3490838385] [v2:172.21.15.131:6818/3490838385,v1:172.21.15.131:6819/3490838385] exists,up c05d9ea6-d3bb-4333-a68a-f2d689e41310 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6826/1397303359,v1:172.21.15.26:6827/1397303359] [v2:172.21.15.26:6828/1397303359,v1:172.21.15.26:6829/1397303359] exists,up 34848530-043d-4447-a445-4c1e4d5f1c8b 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/3721710619,v1:172.21.15.110:6825/3721710619] [v2:172.21.15.110:6826/3721710619,v1:172.21.15.110:6827/3721710619] exists,up d5bcbd75-0ca4-450e-b735-3bcb342f53a9 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6824/2935357051,v1:172.21.15.131:6825/2935357051] [v2:172.21.15.131:6826/2935357051,v1:172.21.15.131:6827/2935357051] exists,up e17072b7-66af-42ed-a40e-7b472fa918f5 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 2.b [8,11] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 2.13 [2,11] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 2.1b [2,4] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 3.3 [4,9] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 3.b [3,9] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 3.e [5,9] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 3.14 [4,6] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:pg_upmap_items 3.17 [4,8] 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/3591553425 expires 2024-04-24T18:08:10.757945+0000 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/2032390504 expires 2024-04-24T18:08:10.757945+0000 2024-04-23T18:16:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6801/2629443403 expires 2024-04-24T18:08:10.757945+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6800/2629443403 expires 2024-04-24T18:08:10.757945+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/4146543668 expires 2024-04-24T18:07:44.350411+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/3418832937 expires 2024-04-24T18:07:44.350411+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/2243122498 expires 2024-04-24T18:07:44.350411+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6800/259946325 expires 2024-04-24T18:07:44.350411+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6801/259946325 expires 2024-04-24T18:07:44.350411+0000 2024-04-23T18:16:55.971 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/1804299815 expires 2024-04-24T18:08:10.757945+0000 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 msgr2=0x7f5ee40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 0x7f5ee40a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5efc102a00 tx=0x7f5eec009260 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 msgr2=0x7f5efc103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000d950 tx=0x7f5ef000dc60 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 shutdown_connections 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f5ee40a3c40 0x7f5ee40a6100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5efc103540 0x7f5efc1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5efc1019a0 0x7f5efc105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 --2- 172.21.15.26:0/3429833875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5efc102ba0 0x7f5efc103f80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 >> 172.21.15.26:0/3429833875 conn(0x7f5efc0fcf40 msgr2=0x7f5efc0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 shutdown_connections 2024-04-23T18:16:55.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:55.977+0000 7f5f010ba700 1 -- 172.21.15.26:0/3429833875 wait complete. 2024-04-23T18:16:56.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:56 smithi026 ceph-mon[29364]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:16:56.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:56 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3429833875' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:16:56.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:56 smithi110 ceph-mon[32585]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:16:56.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:56 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3429833875' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:16:56.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:56 smithi131 ceph-mon[33041]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:16:56.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:56 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3429833875' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:16:56.789 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-23T18:16:56.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 -- 172.21.15.26:0/2760665099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224100e90 msgr2=0x7f22241012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 --2- 172.21.15.26:0/2760665099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224100e90 0x7f22241012b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f220c009900 tx=0x7f220c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 -- 172.21.15.26:0/2760665099 shutdown_connections 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 --2- 172.21.15.26:0/2760665099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 0x7f22241092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 --2- 172.21.15.26:0/2760665099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224101fe0 0x7f2224102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 --2- 172.21.15.26:0/2760665099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224100e90 0x7f22241012b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 -- 172.21.15.26:0/2760665099 >> 172.21.15.26:0/2760665099 conn(0x7f22240fc400 msgr2=0x7f22240fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 -- 172.21.15.26:0/2760665099 shutdown_connections 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.918+0000 7f222a336640 1 -- 172.21.15.26:0/2760665099 wait complete. 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 Processor -- start 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 -- start start 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224100e90 0x7f2224104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 0x7f2224103a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f222410c9e0 con 0x7f2224101fe0 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f222410c860 con 0x7f22241029a0 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f222a336640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f222410cb60 con 0x7f2224100e90 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f22237fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f22237fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60964/0 (socket says 172.21.15.26:60964) 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.919+0000 7f22237fe640 1 -- 172.21.15.26:0/720829377 learned_addr learned my addr 172.21.15.26:0/720829377 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22288d3640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 0x7f2224103a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f2223fff640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224100e90 0x7f2224104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 -- 172.21.15.26:0/720829377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224100e90 msgr2=0x7f2224104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224100e90 0x7f2224104f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 -- 172.21.15.26:0/720829377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 msgr2=0x7f2224103a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 0x7f2224103a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 -- 172.21.15.26:0/720829377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2224104310 con 0x7f2224101fe0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22237fe640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f221400d950 tx=0x7f221400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214017070 con 0x7f2224101fe0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22241a2340 con 0x7f2224101fe0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2214009c30 con 0x7f2224101fe0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221400f620 con 0x7f2224101fe0 2024-04-23T18:16:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.920+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22241a2840 con 0x7f2224101fe0 2024-04-23T18:16:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.921+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22241a2b00 con 0x7f2224101fe0 2024-04-23T18:16:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.923+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f221401a070 con 0x7f2224101fe0 2024-04-23T18:16:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.924+0000 7f22217fa640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 0x7f22100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.924+0000 7f2223fff640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 0x7f22100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.924+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f22140cfd60 con 0x7f2224101fe0 2024-04-23T18:16:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.924+0000 7f2223fff640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 0x7f22100a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f220c000c00 tx=0x7f220c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:56.927+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2214009de0 con 0x7f2224101fe0 2024-04-23T18:16:57.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.117+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22241a31d0 con 0x7f2224101fe0 2024-04-23T18:16:57.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.118+0000 7f22217fa640 1 -- 172.21.15.26:0/720829377 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7f221409c100 con 0x7f2224101fe0 2024-04-23T18:16:57.112 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:57.112 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:57.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 msgr2=0x7f22100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 0x7f22100a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f220c000c00 tx=0x7f220c0314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 msgr2=0x7f2224103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f221400d950 tx=0x7f221400de10 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 shutdown_connections 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f22100a3bb0 0x7f22100a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22241029a0 0x7f2224103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2224101fe0 0x7f2224103540 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 --2- 172.21.15.26:0/720829377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2224100e90 0x7f2224104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 >> 172.21.15.26:0/720829377 conn(0x7f22240fc400 msgr2=0x7f22240690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 shutdown_connections 2024-04-23T18:16:57.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.121+0000 7f222a336640 1 -- 172.21.15.26:0/720829377 wait complete. 2024-04-23T18:16:57.126 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:16:57.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 -- 172.21.15.26:0/1544229665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc102040 msgr2=0x7f41dc1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 --2- 172.21.15.26:0/1544229665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc102040 0x7f41dc1024c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009900 tx=0x7f41d002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 -- 172.21.15.26:0/1544229665 shutdown_connections 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 --2- 172.21.15.26:0/1544229665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 --2- 172.21.15.26:0/1544229665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc102040 0x7f41dc1024c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 --2- 172.21.15.26:0/1544229665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41dc100e60 0x7f41dc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 -- 172.21.15.26:0/1544229665 >> 172.21.15.26:0/1544229665 conn(0x7f41dc0fc440 msgr2=0x7f41dc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 -- 172.21.15.26:0/1544229665 shutdown_connections 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.237+0000 7f41e43ef640 1 -- 172.21.15.26:0/1544229665 wait complete. 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 Processor -- start 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 -- start start 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41dc102040 0x7f41dc19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10ca60 con 0x7f41dc100e60 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10c8e0 con 0x7f41dc102a00 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e43ef640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc10cbe0 con 0x7f41dc102040 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60982/0 (socket says 172.21.15.26:60982) 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 -- 172.21.15.26:0/2215819362 learned_addr learned my addr 172.21.15.26:0/2215819362 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e298c640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 -- 172.21.15.26:0/2215819362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41dc102040 msgr2=0x7f41dc19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41dc102040 0x7f41dc19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 -- 172.21.15.26:0/2215819362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 msgr2=0x7f41dc19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e218b640 1 -- 172.21.15.26:0/2215819362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc19fe60 con 0x7f41dc100e60 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.238+0000 7f41e298c640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:57.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41e218b640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c970 tx=0x7f41d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4007c10 con 0x7f41dc100e60 2024-04-23T18:16:57.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41d4007db0 con 0x7f41dc100e60 2024-04-23T18:16:57.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc1a00f0 con 0x7f41dc100e60 2024-04-23T18:16:57.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d400f870 con 0x7f41dc100e60 2024-04-23T18:16:57.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.239+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1a0710 con 0x7f41dc100e60 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.240+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc108200 con 0x7f41dc100e60 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.241+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f41d400fa10 con 0x7f41dc100e60 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.242+0000 7f41db7fe640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 0x7f41c40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.242+0000 7f41e198a640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 0x7f41c40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.242+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f41d40d0810 con 0x7f41dc100e60 2024-04-23T18:16:57.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.242+0000 7f41e198a640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 0x7f41c40a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f41d00096c0 tx=0x7f41d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.246+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41d409d230 con 0x7f41dc100e60 2024-04-23T18:16:57.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.432+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f41dc062b80 con 0x7f41dc100e60 2024-04-23T18:16:57.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.432+0000 7f41db7fe640 1 -- 172.21.15.26:0/2215819362 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f41d409cb30 con 0x7f41dc100e60 2024-04-23T18:16:57.426 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:16:57.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 msgr2=0x7f41c40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 0x7f41c40a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f41d00096c0 tx=0x7f41d0000bc0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 msgr2=0x7f41dc06b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c970 tx=0x7f41d400ce30 comp rx=0 tx=0).stop 2024-04-23T18:16:57.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 shutdown_connections 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f41c40a3c30 0x7f41c40a60f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41dc102a00 0x7f41dc19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41dc102040 0x7f41dc19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 --2- 172.21.15.26:0/2215819362 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f41dc100e60 0x7f41dc06b0c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 >> 172.21.15.26:0/2215819362 conn(0x7f41dc0fc440 msgr2=0x7f41dc1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 shutdown_connections 2024-04-23T18:16:57.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.434+0000 7f41e43ef640 1 -- 172.21.15.26:0/2215819362 wait complete. 2024-04-23T18:16:57.439 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 -- 172.21.15.26:0/2309702577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 msgr2=0x7fc254101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 --2- 172.21.15.26:0/2309702577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc254101640 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc244009900 tx=0x7fc24402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 -- 172.21.15.26:0/2309702577 shutdown_connections 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 --2- 172.21.15.26:0/2309702577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 0x7fc254102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 --2- 172.21.15.26:0/2309702577 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc254101640 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 --2- 172.21.15.26:0/2309702577 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc254100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 -- 172.21.15.26:0/2309702577 >> 172.21.15.26:0/2309702577 conn(0x7fc2540fc440 msgr2=0x7fc2540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 -- 172.21.15.26:0/2309702577 shutdown_connections 2024-04-23T18:16:57.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.551+0000 7fc25920b640 1 -- 172.21.15.26:0/2309702577 wait complete. 2024-04-23T18:16:57.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 Processor -- start 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 -- start start 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc25419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 0x7fc2541a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410fd00 con 0x7fc2541011e0 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410fb80 con 0x7fc254101b80 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25920b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410fe80 con 0x7fc254100880 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:32776/0 (socket says 172.21.15.26:32776) 2024-04-23T18:16:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25259c640 1 -- 172.21.15.26:0/3343900482 learned_addr learned my addr 172.21.15.26:0/3343900482 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.552+0000 7fc25359e640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 0x7fc2541a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc252d9d640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc25419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 -- 172.21.15.26:0/3343900482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 msgr2=0x7fc25419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc25419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 -- 172.21.15.26:0/3343900482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 msgr2=0x7fc2541a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 0x7fc2541a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 -- 172.21.15.26:0/3343900482 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2541a1f50 con 0x7fc2541011e0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc252d9d640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc25419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25259c640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc244031500 tx=0x7fc2440043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24403f030 con 0x7fc2541011e0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2541a21e0 con 0x7fc2541011e0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc244003800 con 0x7fc2541011e0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244035070 con 0x7fc2541011e0 2024-04-23T18:16:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.553+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2541a2720 con 0x7fc2541011e0 2024-04-23T18:16:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.554+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc25410b770 con 0x7fc2541011e0 2024-04-23T18:16:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.555+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc2440039b0 con 0x7fc2541011e0 2024-04-23T18:16:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.556+0000 7fc23bfff640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 0x7fc23c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.556+0000 7fc252d9d640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 0x7fc23c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.556+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc2440f1460 con 0x7fc2541011e0 2024-04-23T18:16:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.556+0000 7fc252d9d640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 0x7fc23c0a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc2480058c0 tx=0x7fc248005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.560+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2440bde50 con 0x7fc2541011e0 2024-04-23T18:16:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.750+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc25402d7c0 con 0x7fc2541011e0 2024-04-23T18:16:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.750+0000 7fc23bfff640 1 -- 172.21.15.26:0/3343900482 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fc2440bd750 con 0x7fc2541011e0 2024-04-23T18:16:57.744 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:16:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 msgr2=0x7fc23c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 0x7fc23c0a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc2480058c0 tx=0x7fc248005d30 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 msgr2=0x7fc25419a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc244031500 tx=0x7fc2440043c0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 shutdown_connections 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc23c0a3bb0 0x7fc23c0a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc254101b80 0x7fc2541a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc2541011e0 0x7fc25419a870 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 --2- 172.21.15.26:0/3343900482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc254100880 0x7fc25419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 >> 172.21.15.26:0/3343900482 conn(0x7fc2540fc440 msgr2=0x7fc2540fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 shutdown_connections 2024-04-23T18:16:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.753+0000 7fc25920b640 1 -- 172.21.15.26:0/3343900482 wait complete. 2024-04-23T18:16:57.757 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 -- 172.21.15.26:0/3211973074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c100e60 msgr2=0x7f509c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 --2- 172.21.15.26:0/3211973074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c100e60 0x7f509c101280 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5098009900 tx=0x7f509802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 -- 172.21.15.26:0/3211973074 shutdown_connections 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 --2- 172.21.15.26:0/3211973074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c102040 0x7f509c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 --2- 172.21.15.26:0/3211973074 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102a00 0x7f509c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 --2- 172.21.15.26:0/3211973074 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c100e60 0x7f509c101280 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 -- 172.21.15.26:0/3211973074 >> 172.21.15.26:0/3211973074 conn(0x7f509c0fc440 msgr2=0x7f509c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 -- 172.21.15.26:0/3211973074 shutdown_connections 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.869+0000 7f50a47d8640 1 -- 172.21.15.26:0/3211973074 wait complete. 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 Processor -- start 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 -- start start 2024-04-23T18:16:57.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c100e60 0x7f509c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102040 0x7f509c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f509c10db30 con 0x7f509c102a00 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f509c10d9b0 con 0x7f509c100e60 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a47d8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f509c10dcb0 con 0x7f509c102040 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:32798/0 (socket says 172.21.15.26:32798) 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 -- 172.21.15.26:0/2517503773 learned_addr learned my addr 172.21.15.26:0/2517503773 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a1d73640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102040 0x7f509c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2574640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c100e60 0x7f509c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 -- 172.21.15.26:0/2517503773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102040 msgr2=0x7f509c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102040 0x7f509c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 -- 172.21.15.26:0/2517503773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c100e60 msgr2=0x7f509c1982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c100e60 0x7f509c1982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 -- 172.21.15.26:0/2517503773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c19fe70 con 0x7f509c102a00 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50a2d75640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f509400cad0 tx=0x7f509400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.870+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509400ae20 con 0x7f509c102a00 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.871+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5094005020 con 0x7f509c102a00 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.871+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509c1a0160 con 0x7f509c102a00 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.871+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509400f6b0 con 0x7f509c102a00 2024-04-23T18:16:57.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.871+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509c1a06b0 con 0x7f509c102a00 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.872+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509c108320 con 0x7f509c102a00 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.873+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f50940051d0 con 0x7f509c102a00 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.873+0000 7f50937fe640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 0x7f50880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.874+0000 7f50a2574640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 0x7f50880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.874+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f50940d04b0 con 0x7f509c102a00 2024-04-23T18:16:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.874+0000 7f50a2574640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 0x7f50880a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f50980314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:57.878+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5094014030 con 0x7f509c102a00 2024-04-23T18:16:58.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.068+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f509c062b50 con 0x7f509c102a00 2024-04-23T18:16:58.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.068+0000 7f50937fe640 1 -- 172.21.15.26:0/2517503773 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f509409cea0 con 0x7f509c102a00 2024-04-23T18:16:58.062 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 msgr2=0x7f50880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 0x7f50880a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f50980314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 msgr2=0x7f509c19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f509400cad0 tx=0x7f509400cf90 comp rx=0 tx=0).stop 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 shutdown_connections 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f50880a3bb0 0x7f50880a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f509c102a00 0x7f509c19f800 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f509c102040 0x7f509c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 --2- 172.21.15.26:0/2517503773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509c100e60 0x7f509c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 >> 172.21.15.26:0/2517503773 conn(0x7f509c0fc440 msgr2=0x7f509c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 shutdown_connections 2024-04-23T18:16:58.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.070+0000 7f50a47d8640 1 -- 172.21.15.26:0/2517503773 wait complete. 2024-04-23T18:16:58.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.184+0000 7fb900031640 1 -- 172.21.15.26:0/759761141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 msgr2=0x7fb8f8100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.184+0000 7fb900031640 1 --2- 172.21.15.26:0/759761141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f8100ca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009ca0 tx=0x7fb8f402a950 comp rx=0 tx=0).stop 2024-04-23T18:16:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.184+0000 7fb900031640 1 -- 172.21.15.26:0/759761141 shutdown_connections 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.184+0000 7fb900031640 1 --2- 172.21.15.26:0/759761141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f8101b80 0x7fb8f8102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 --2- 172.21.15.26:0/759761141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f81011e0 0x7fb8f8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 --2- 172.21.15.26:0/759761141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f8100ca0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- 172.21.15.26:0/759761141 >> 172.21.15.26:0/759761141 conn(0x7fb8f80fc400 msgr2=0x7fb8f80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- 172.21.15.26:0/759761141 shutdown_connections 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- 172.21.15.26:0/759761141 wait complete. 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 Processor -- start 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- start start 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 0x7fb8f819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 0x7fb8f81a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f810fcd0 con 0x7fb8f8100880 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f810fb50 con 0x7fb8f81011e0 2024-04-23T18:16:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb900031640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f810fe50 con 0x7fb8f8101b80 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb8fddcd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb8fddcd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48620/0 (socket says 172.21.15.26:48620) 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.185+0000 7fb8fddcd640 1 -- 172.21.15.26:0/960642057 learned_addr learned my addr 172.21.15.26:0/960642057 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fd5cc640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 0x7fb8f819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fe5ce640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 0x7fb8f81a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 -- 172.21.15.26:0/960642057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 msgr2=0x7fb8f81a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 0x7fb8f81a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 -- 172.21.15.26:0/960642057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 msgr2=0x7fb8f819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 0x7fb8f819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 -- 172.21.15.26:0/960642057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f81a1f80 con 0x7fb8f8100880 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fd5cc640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 0x7fb8f819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fe5ce640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 0x7fb8f81a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8fddcd640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000f80 tx=0x7fb8f4031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4034040 con 0x7fb8f8100880 2024-04-23T18:16:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f81a2210 con 0x7fb8f8100880 2024-04-23T18:16:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8f4004040 con 0x7fb8f8100880 2024-04-23T18:16:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40456e0 con 0x7fb8f8100880 2024-04-23T18:16:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.186+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f81a2700 con 0x7fb8f8100880 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.187+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8f810b750 con 0x7fb8f8100880 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.188+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb8f4037020 con 0x7fb8f8100880 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.189+0000 7fb8eeffd640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 0x7fb8e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.189+0000 7fb8fd5cc640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 0x7fb8e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.189+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb8f40f17e0 con 0x7fb8f8100880 2024-04-23T18:16:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.189+0000 7fb8fd5cc640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 0x7fb8e40a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb8f819b870 tx=0x7fb8e800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.193+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8f40be170 con 0x7fb8f8100880 2024-04-23T18:16:58.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.419+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8f802d5c0 con 0x7fb8f8100880 2024-04-23T18:16:58.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.420+0000 7fb8eeffd640 1 -- 172.21.15.26:0/960642057 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fb8f40bda70 con 0x7fb8f8100880 2024-04-23T18:16:58.414 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:58.414 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:16:58.414 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:16:58.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 msgr2=0x7fb8e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 0x7fb8e40a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb8f819b870 tx=0x7fb8e800a5c0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 msgr2=0x7fb8f819a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000f80 tx=0x7fb8f4031cb0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 shutdown_connections 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb8e40a3bb0 0x7fb8e40a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f81011e0 0x7fb8f819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8f8101b80 0x7fb8f81a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 --2- 172.21.15.26:0/960642057 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8f8100880 0x7fb8f819a360 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 >> 172.21.15.26:0/960642057 conn(0x7fb8f80fc400 msgr2=0x7fb8f80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 shutdown_connections 2024-04-23T18:16:58.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.422+0000 7fb900031640 1 -- 172.21.15.26:0/960642057 wait complete. 2024-04-23T18:16:58.425 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:58 smithi026 ceph-mon[29364]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:16:58.425 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:58 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/720829377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:58.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:58 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2215819362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:58 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3343900482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:58.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:58 smithi110 ceph-mon[32585]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:16:58.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:58 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/720829377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:58.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:58 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2215819362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:58 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3343900482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:58 smithi131 ceph-mon[33041]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-23T18:16:58.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:58 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/720829377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:58.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:58 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2215819362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:58 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3343900482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:16:58.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- 172.21.15.26:0/3434464032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c103b70 msgr2=0x7f0b0c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/3434464032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c103b70 0x7f0b0c10a400 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009ca0 tx=0x7f0b0402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- 172.21.15.26:0/3434464032 shutdown_connections 2024-04-23T18:16:58.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/3434464032 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c103b70 0x7f0b0c10a400 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/3434464032 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/3434464032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b0c101fc0 0x7f0b0c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- 172.21.15.26:0/3434464032 >> 172.21.15.26:0/3434464032 conn(0x7f0b0c0fd550 msgr2=0x7f0b0c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- 172.21.15.26:0/3434464032 shutdown_connections 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- 172.21.15.26:0/3434464032 wait complete. 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 Processor -- start 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 -- start start 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.537+0000 7f0b12e3c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c101fc0 0x7f0b0c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b0c103b70 0x7f0b0c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c10db00 con 0x7f0b0c1031b0 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c10d980 con 0x7f0b0c103b70 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c10dc80 con 0x7f0b0c101fc0 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48640/0 (socket says 172.21.15.26:48640) 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 -- 172.21.15.26:0/4199194399 learned_addr learned my addr 172.21.15.26:0/4199194399 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b10bd8640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c101fc0 0x7f0b0c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 -- 172.21.15.26:0/4199194399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c101fc0 msgr2=0x7f0b0c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c101fc0 0x7f0b0c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 -- 172.21.15.26:0/4199194399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b0c103b70 msgr2=0x7f0b0c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b0c103b70 0x7f0b0c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 -- 172.21.15.26:0/4199194399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0c19fd80 con 0x7f0b0c1031b0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b0bfff640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d950 tx=0x7f0af800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8017070 con 0x7f0b0c1031b0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af8009c30 con 0x7f0b0c1031b0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0c1a0070 con 0x7f0b0c1031b0 2024-04-23T18:16:58.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af800f620 con 0x7f0b0c1031b0 2024-04-23T18:16:58.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.538+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b0c1a05c0 con 0x7f0b0c1031b0 2024-04-23T18:16:58.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.540+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b0c109330 con 0x7f0b0c1031b0 2024-04-23T18:16:58.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.541+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0af8010460 con 0x7f0b0c1031b0 2024-04-23T18:16:58.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.541+0000 7f0b09ffb640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 0x7f0afc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.541+0000 7f0b10bd8640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 0x7f0afc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.541+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0af80d00d0 con 0x7f0b0c1031b0 2024-04-23T18:16:58.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.542+0000 7f0b10bd8640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 0x7f0afc0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b00005fd0 tx=0x7f0b00005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.546+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0af809cac0 con 0x7f0b0c1031b0 2024-04-23T18:16:58.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.733+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b0c062bf0 con 0x7f0b0c1031b0 2024-04-23T18:16:58.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.734+0000 7f0b09ffb640 1 -- 172.21.15.26:0/4199194399 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7f0af8009de0 con 0x7f0b0c1031b0 2024-04-23T18:16:58.728 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:16:58.728 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 msgr2=0x7f0afc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 0x7f0afc0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b00005fd0 tx=0x7f0b00005850 comp rx=0 tx=0).stop 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 msgr2=0x7f0b0c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0af800d950 tx=0x7f0af800de10 comp rx=0 tx=0).stop 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 shutdown_connections 2024-04-23T18:16:58.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f0afc0a3bb0 0x7f0afc0a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b0c103b70 0x7f0b0c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0b0c1031b0 0x7f0b0c1986a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 --2- 172.21.15.26:0/4199194399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b0c101fc0 0x7f0b0c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 >> 172.21.15.26:0/4199194399 conn(0x7f0b0c0fd550 msgr2=0x7f0b0c106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 shutdown_connections 2024-04-23T18:16:58.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.736+0000 7f0b12e3c640 1 -- 172.21.15.26:0/4199194399 wait complete. 2024-04-23T18:16:58.741 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:16:58.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 -- 172.21.15.26:0/499219302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68102040 msgr2=0x7f4e681024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/499219302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68102040 0x7f4e681024c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009900 tx=0x7f4e5002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 -- 172.21.15.26:0/499219302 shutdown_connections 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/499219302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e68102a00 0x7f4e68109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/499219302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68102040 0x7f4e681024c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/499219302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68100e60 0x7f4e68101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 -- 172.21.15.26:0/499219302 >> 172.21.15.26:0/499219302 conn(0x7f4e680fc440 msgr2=0x7f4e680fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 -- 172.21.15.26:0/499219302 shutdown_connections 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 -- 172.21.15.26:0/499219302 wait complete. 2024-04-23T18:16:58.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.853+0000 7f4e6eee8640 1 Processor -- start 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 -- start start 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68102040 0x7f4e681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e68102a00 0x7f4e6819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810db30 con 0x7f4e68100e60 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810d9b0 con 0x7f4e68102a00 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810dcb0 con 0x7f4e68102040 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48662/0 (socket says 172.21.15.26:48662) 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 -- 172.21.15.26:0/2195756841 learned_addr learned my addr 172.21.15.26:0/2195756841 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e67fff640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68102040 0x7f4e681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 -- 172.21.15.26:0/2195756841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68102040 msgr2=0x7f4e681986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68102040 0x7f4e681986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 -- 172.21.15.26:0/2195756841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e68102a00 msgr2=0x7f4e6819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e68102a00 0x7f4e6819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 -- 172.21.15.26:0/2195756841 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6819fdd0 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6cc84640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800d950 tx=0x7f4e5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e58017070 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e58009c30 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5800f620 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.854+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e681a00c0 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.855+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e681a05c0 con 0x7f4e68100e60 2024-04-23T18:16:58.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.855+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e68108200 con 0x7f4e68100e60 2024-04-23T18:16:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.857+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4e5801a070 con 0x7f4e68100e60 2024-04-23T18:16:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.857+0000 7f4e65ffb640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 0x7f4e540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.858+0000 7f4e67fff640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 0x7f4e540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.858+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f4e580cfee0 con 0x7f4e68100e60 2024-04-23T18:16:58.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.858+0000 7f4e67fff640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 0x7f4e540a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e5002a930 tx=0x7f4e50000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:58.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:58.862+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e58009de0 con 0x7f4e68100e60 2024-04-23T18:16:59.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.052+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e6802d5c0 con 0x7f4e68100e60 2024-04-23T18:16:59.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.052+0000 7f4e65ffb640 1 -- 172.21.15.26:0/2195756841 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f4e6802d5c0 con 0x7f4e68100e60 2024-04-23T18:16:59.046 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:16:59.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.054+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 msgr2=0x7f4e540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.054+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 0x7f4e540a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e5002a930 tx=0x7f4e50000bc0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.054+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 msgr2=0x7f4e681981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.054+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800d950 tx=0x7f4e5800de10 comp rx=0 tx=0).stop 2024-04-23T18:16:59.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 shutdown_connections 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f4e540a3bb0 0x7f4e540a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e68102a00 0x7f4e6819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e68102040 0x7f4e681986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 --2- 172.21.15.26:0/2195756841 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f4e68100e60 0x7f4e681981b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 >> 172.21.15.26:0/2195756841 conn(0x7f4e680fc440 msgr2=0x7f4e681053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 shutdown_connections 2024-04-23T18:16:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.055+0000 7f4e6eee8640 1 -- 172.21.15.26:0/2195756841 wait complete. 2024-04-23T18:16:59.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:16:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- 172.21.15.26:0/2845592299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c1021e0 msgr2=0x7fb66c108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 --2- 172.21.15.26:0/2845592299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c1021e0 0x7fb66c108a70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb664009ca0 tx=0x7fb66402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- 172.21.15.26:0/2845592299 shutdown_connections 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 --2- 172.21.15.26:0/2845592299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c1021e0 0x7fb66c108a70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 --2- 172.21.15.26:0/2845592299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 0x7fb66c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 --2- 172.21.15.26:0/2845592299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c100630 0x7fb66c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- 172.21.15.26:0/2845592299 >> 172.21.15.26:0/2845592299 conn(0x7fb66c0fbba0 msgr2=0x7fb66c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- 172.21.15.26:0/2845592299 shutdown_connections 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- 172.21.15.26:0/2845592299 wait complete. 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 Processor -- start 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.172+0000 7fb6742bb640 1 -- start start 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 0x7fb66c19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c1021e0 0x7fb66c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10c1e0 con 0x7fb66c100630 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10c060 con 0x7fb66c1021e0 2024-04-23T18:16:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10c360 con 0x7fb66c101820 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48682/0 (socket says 172.21.15.26:48682) 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 -- 172.21.15.26:0/1406138722 learned_addr learned my addr 172.21.15.26:0/1406138722 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb671856640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 0x7fb66c19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672858640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c1021e0 0x7fb66c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 -- 172.21.15.26:0/1406138722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 msgr2=0x7fb66c19cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 0x7fb66c19cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 -- 172.21.15.26:0/1406138722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c1021e0 msgr2=0x7fb66c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c1021e0 0x7fb66c19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 -- 172.21.15.26:0/1406138722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c19fd30 con 0x7fb66c100630 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb672057640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb66800b450 tx=0x7fb66800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668015020 con 0x7fb66c100630 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb668011d40 con 0x7fb66c100630 2024-04-23T18:16:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c19ffc0 con 0x7fb66c100630 2024-04-23T18:16:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66800f340 con 0x7fb66c100630 2024-04-23T18:16:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.173+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c1a0530 con 0x7fb66c100630 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.174+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66c1a07f0 con 0x7fb66c100630 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.176+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb668018020 con 0x7fb66c100630 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.176+0000 7fb6637fe640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 0x7fb6580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.176+0000 7fb671856640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 0x7fb6580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.176+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb6680d0510 con 0x7fb66c100630 2024-04-23T18:16:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.177+0000 7fb671856640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 0x7fb6580a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005e10 tx=0x7fb65c005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.180+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb66801f020 con 0x7fb66c100630 2024-04-23T18:16:59.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.367+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb66c107c80 con 0x7fb66c100630 2024-04-23T18:16:59.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.367+0000 7fb6637fe640 1 -- 172.21.15.26:0/1406138722 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fb66800d040 con 0x7fb66c100630 2024-04-23T18:16:59.361 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:16:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 msgr2=0x7fb6580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 0x7fb6580a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005e10 tx=0x7fb65c005da0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 msgr2=0x7fb66c0fdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb66800b450 tx=0x7fb66800b910 comp rx=0 tx=0).stop 2024-04-23T18:16:59.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 shutdown_connections 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb6580a3bb0 0x7fb6580a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb66c1021e0 0x7fb66c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb66c101820 0x7fb66c19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 --2- 172.21.15.26:0/1406138722 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb66c100630 0x7fb66c0fdc30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 >> 172.21.15.26:0/1406138722 conn(0x7fb66c0fbba0 msgr2=0x7fb66c104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.369+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 shutdown_connections 2024-04-23T18:16:59.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.370+0000 7fb6742bb640 1 -- 172.21.15.26:0/1406138722 wait complete. 2024-04-23T18:16:59.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:59 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2517503773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:59 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/960642057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:59 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4199194399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:16:59 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2195756841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:16:59.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:59 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2517503773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:59 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/960642057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:59 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4199194399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:16:59 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2195756841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3543050241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08101fe0 msgr2=0x7fde08102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3543050241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08101fe0 0x7fde08102460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009900 tx=0x7fddf802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3543050241 shutdown_connections 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3543050241 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 0x7fde081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3543050241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08101fe0 0x7fde08102460 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3543050241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08100e90 0x7fde081012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.485+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3543050241 >> 172.21.15.26:0/3543050241 conn(0x7fde080fc400 msgr2=0x7fde080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3543050241 shutdown_connections 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3543050241 wait complete. 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 Processor -- start 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- start start 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08101fe0 0x7fde0819d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 0x7fde0819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810cac0 con 0x7fde08100e90 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810c940 con 0x7fde08101fe0 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0f5f9640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810cc40 con 0x7fde081029a0 2024-04-23T18:16:59.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0d395640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0d395640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48698/0 (socket says 172.21.15.26:48698) 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0d395640 1 -- 172.21.15.26:0/3467666128 learned_addr learned my addr 172.21.15.26:0/3467666128 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.486+0000 7fde0cb94640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08101fe0 0x7fde0819d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 -- 172.21.15.26:0/3467666128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 msgr2=0x7fde0819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0db96640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 0x7fde0819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 0x7fde0819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 -- 172.21.15.26:0/3467666128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08101fe0 msgr2=0x7fde0819d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08101fe0 0x7fde0819d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 -- 172.21.15.26:0/3467666128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0819ff70 con 0x7fde08100e90 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0d395640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde0400b4a0 tx=0x7fde0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04014070 con 0x7fde08100e90 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde04010ac0 con 0x7fde08100e90 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a0260 con 0x7fde08100e90 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04017c30 con 0x7fde08100e90 2024-04-23T18:16:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.487+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a0850 con 0x7fde08100e90 2024-04-23T18:16:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.488+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde08108330 con 0x7fde08100e90 2024-04-23T18:16:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.489+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fde04017070 con 0x7fde08100e90 2024-04-23T18:16:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.490+0000 7fddfe7fc640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 0x7fddf40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.490+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fde040cfa70 con 0x7fde08100e90 2024-04-23T18:16:59.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.490+0000 7fde0cb94640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 0x7fddf40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.490+0000 7fde0cb94640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 0x7fddf40a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009230 tx=0x7fddf80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.493+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde0409c460 con 0x7fde08100e90 2024-04-23T18:16:59.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:59 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2517503773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:59 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/960642057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:59 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4199194399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:59.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:16:59 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2195756841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:59.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.682+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde0802d830 con 0x7fde08100e90 2024-04-23T18:16:59.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.683+0000 7fddfe7fc640 1 -- 172.21.15.26:0/3467666128 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fde0409bd60 con 0x7fde08100e90 2024-04-23T18:16:59.677 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 msgr2=0x7fddf40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 0x7fddf40a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009230 tx=0x7fddf80314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 msgr2=0x7fde08069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde0400b4a0 tx=0x7fde0400b960 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 shutdown_connections 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fddf40a3c30 0x7fddf40a60f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde08101fe0 0x7fde0819d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde081029a0 0x7fde0819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 --2- 172.21.15.26:0/3467666128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fde08100e90 0x7fde08069c70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 >> 172.21.15.26:0/3467666128 conn(0x7fde080fc400 msgr2=0x7fde08105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 shutdown_connections 2024-04-23T18:16:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.685+0000 7fde0f5f9640 1 -- 172.21.15.26:0/3467666128 wait complete. 2024-04-23T18:16:59.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 -- 172.21.15.26:0/3204844096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390105410 msgr2=0x7fc390107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 --2- 172.21.15.26:0/3204844096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390105410 0x7fc390107800 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc380009900 tx=0x7fc38002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 -- 172.21.15.26:0/3204844096 shutdown_connections 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 --2- 172.21.15.26:0/3204844096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390107d40 0x7fc39010a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 --2- 172.21.15.26:0/3204844096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390105410 0x7fc390107800 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 --2- 172.21.15.26:0/3204844096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390102ae0 0x7fc390104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 -- 172.21.15.26:0/3204844096 >> 172.21.15.26:0/3204844096 conn(0x7fc3900fc430 msgr2=0x7fc3900fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 -- 172.21.15.26:0/3204844096 shutdown_connections 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.798+0000 7fc397a3c640 1 -- 172.21.15.26:0/3204844096 wait complete. 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 Processor -- start 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 -- start start 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 0x7fc390198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 0x7fc390198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc39010daf0 con 0x7fc390107d40 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc39010d970 con 0x7fc390102ae0 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc397a3c640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc39010dc70 con 0x7fc390105410 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48706/0 (socket says 172.21.15.26:48706) 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 -- 172.21.15.26:0/4207019977 learned_addr learned my addr 172.21.15.26:0/4207019977 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc394fd7640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 0x7fc390198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc3957d8640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 0x7fc390198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 -- 172.21.15.26:0/4207019977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 msgr2=0x7fc390198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 0x7fc390198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 -- 172.21.15.26:0/4207019977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 msgr2=0x7fc390198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 0x7fc390198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc395fd9640 1 -- 172.21.15.26:0/4207019977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39019fe60 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc394fd7640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 0x7fc390198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.799+0000 7fc3957d8640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 0x7fc390198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc395fd9640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc38800eaa0 tx=0x7fc38800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38800cdb0 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc388005020 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3901a0150 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3880116d0 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.800+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3901a0670 con 0x7fc390107d40 2024-04-23T18:16:59.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.801+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3900fdf20 con 0x7fc390107d40 2024-04-23T18:16:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.802+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc3880051d0 con 0x7fc390107d40 2024-04-23T18:16:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.803+0000 7fc3867fc640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 0x7fc37c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.803+0000 7fc3957d8640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 0x7fc37c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.803+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc3880d0a20 con 0x7fc390107d40 2024-04-23T18:16:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.803+0000 7fc3957d8640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 0x7fc37c0a60f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0058c0 tx=0x7fc38c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:59.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:16:59.806+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc388016030 con 0x7fc390107d40 2024-04-23T18:17:00.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.032+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc39002d5c0 con 0x7fc390107d40 2024-04-23T18:17:00.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.033+0000 7fc3867fc640 1 -- 172.21.15.26:0/4207019977 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fc38809d430 con 0x7fc390107d40 2024-04-23T18:17:00.027 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:00.027 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:15:11.649607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:17:00.027 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 21 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 msgr2=0x7fc37c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 0x7fc37c0a60f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0058c0 tx=0x7fc38c005d30 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 msgr2=0x7fc39019f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc38800eaa0 tx=0x7fc38800ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 shutdown_connections 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fc37c0a3c30 0x7fc37c0a60f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc390107d40 0x7fc39019f7f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc390105410 0x7fc390198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 --2- 172.21.15.26:0/4207019977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc390102ae0 0x7fc390198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 >> 172.21.15.26:0/4207019977 conn(0x7fc3900fc430 msgr2=0x7fc390103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 shutdown_connections 2024-04-23T18:17:00.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.035+0000 7fc397a3c640 1 -- 172.21.15.26:0/4207019977 wait complete. 2024-04-23T18:17:00.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 -- 172.21.15.26:0/4094050688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8102a30 msgr2=0x7f07c8104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 --2- 172.21.15.26:0/4094050688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8102a30 0x7f07c8104e20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009900 tx=0x7f07b802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 -- 172.21.15.26:0/4094050688 shutdown_connections 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 --2- 172.21.15.26:0/4094050688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8107c90 0x7f07c810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 --2- 172.21.15.26:0/4094050688 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8105360 0x7f07c8107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 --2- 172.21.15.26:0/4094050688 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8102a30 0x7f07c8104e20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 -- 172.21.15.26:0/4094050688 >> 172.21.15.26:0/4094050688 conn(0x7f07c80fc440 msgr2=0x7f07c80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 -- 172.21.15.26:0/4094050688 shutdown_connections 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.151+0000 7f07cd794640 1 -- 172.21.15.26:0/4094050688 wait complete. 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 Processor -- start 2024-04-23T18:17:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 -- start start 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8102a30 0x7f07c81023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8107c90 0x7f07c8100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f07c810db40 con 0x7f07c8105360 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07c810d9c0 con 0x7f07c8102a30 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07cd794640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f07c810dcc0 con 0x7f07c8107c90 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48730/0 (socket says 172.21.15.26:48730) 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 -- 172.21.15.26:0/4031640523 learned_addr learned my addr 172.21.15.26:0/4031640523 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c6ffd640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8102a30 0x7f07c81023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c77fe640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8107c90 0x7f07c8100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 -- 172.21.15.26:0/4031640523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8107c90 msgr2=0x7f07c8100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8107c90 0x7f07c8100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 -- 172.21.15.26:0/4031640523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8102a30 msgr2=0x7f07c81023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8102a30 0x7f07c81023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.152+0000 7f07c67fc640 1 -- 172.21.15.26:0/4031640523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c81017b0 con 0x7f07c8105360 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07c67fc640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00d770 tx=0x7f07bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc017070 con 0x7f07c8105360 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07bc009c30 con 0x7f07c8105360 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c81a2340 con 0x7f07c8105360 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00f620 con 0x7f07c8105360 2024-04-23T18:17:00.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.153+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c81a2890 con 0x7f07c8105360 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.154+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07c80fdf30 con 0x7f07c8105360 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.155+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f07bc01a070 con 0x7f07c8105360 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.156+0000 7f07affff640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 0x7f07b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.156+0000 7f07c6ffd640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 0x7f07b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.156+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f07bc0cff80 con 0x7f07c8105360 2024-04-23T18:17:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.156+0000 7f07c6ffd640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 0x7f07b00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f07b8000c00 tx=0x7f07b80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.159+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07bc009de0 con 0x7f07c8105360 2024-04-23T18:17:00.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.348+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07c802d810 con 0x7f07c8105360 2024-04-23T18:17:00.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.349+0000 7f07affff640 1 -- 172.21.15.26:0/4031640523 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22663 (secure 0 0 0) 0x7f07bc09c970 con 0x7f07c8105360 2024-04-23T18:17:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:00.343 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":88,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:16:12.288554+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 msgr2=0x7f07b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 0x7f07b00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f07b8000c00 tx=0x7f07b80314a0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 msgr2=0x7f07c81009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00d770 tx=0x7f07bc00dc30 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 shutdown_connections 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f07b00a3bb0 0x7f07b00a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f07c8107c90 0x7f07c8100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f07c8105360 0x7f07c81009e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 --2- 172.21.15.26:0/4031640523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07c8102a30 0x7f07c81023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 >> 172.21.15.26:0/4031640523 conn(0x7f07c80fc440 msgr2=0x7f07c8103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 shutdown_connections 2024-04-23T18:17:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.352+0000 7f07cd794640 1 -- 172.21.15.26:0/4031640523 wait complete. 2024-04-23T18:17:00.355 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:00 smithi026 ceph-mon[29364]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:00.356 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:00 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1406138722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.356 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:00 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3467666128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.356 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:00 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4207019977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:00.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:15:06.048360+0000', 'modified': '2024-04-23T18:15:11.649607+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24464, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24433': {'gid': 24433, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6835/2220145154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6834', 'nonce': 2220145154}, {'type': 'v1', 'addr': '172.21.15.131:6835', 'nonce': 2220145154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/3666168856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 3666168856}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 3666168856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6837/2204987461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6836', 'nonce': 2204987461}, {'type': 'v1', 'addr': '172.21.15.131:6837', 'nonce': 2204987461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.131:6839/4056718895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6838', 'nonce': 4056718895}, {'type': 'v1', 'addr': '172.21.15.131:6839', 'nonce': 4056718895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.110:6837/3430274697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 3430274697}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 3430274697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6839/3619972803', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 3619972803}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 3619972803}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24464, 24503, 24460]} 2024-04-23T18:17:00.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:17:00.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:17:00.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:00 smithi110 ceph-mon[32585]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:00.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:00 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1406138722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:00 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3467666128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:00 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4207019977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 -- 172.21.15.26:0/1227730773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20102040 msgr2=0x7f9a201024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 --2- 172.21.15.26:0/1227730773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20102040 0x7f9a201024c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009900 tx=0x7f9a1c02a410 comp rx=0 tx=0).stop 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 -- 172.21.15.26:0/1227730773 shutdown_connections 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 --2- 172.21.15.26:0/1227730773 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a20102a00 0x7f9a20109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 --2- 172.21.15.26:0/1227730773 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20102040 0x7f9a201024c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 --2- 172.21.15.26:0/1227730773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20100e60 0x7f9a20101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 -- 172.21.15.26:0/1227730773 >> 172.21.15.26:0/1227730773 conn(0x7f9a200fc440 msgr2=0x7f9a200fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 -- 172.21.15.26:0/1227730773 shutdown_connections 2024-04-23T18:17:00.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.469+0000 7f9a28508640 1 -- 172.21.15.26:0/1227730773 wait complete. 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 Processor -- start 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 -- start start 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20102040 0x7f9a201975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a20102a00 0x7f9a2019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010ca00 con 0x7f9a20100e60 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010c880 con 0x7f9a20102040 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a28508640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010cb80 con 0x7f9a20102a00 2024-04-23T18:17:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48742/0 (socket says 172.21.15.26:48742) 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 -- 172.21.15.26:0/26166985 learned_addr learned my addr 172.21.15.26:0/26166985 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20102040 0x7f9a201975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 -- 172.21.15.26:0/26166985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a20102a00 msgr2=0x7f9a2019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a20102a00 0x7f9a2019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 -- 172.21.15.26:0/26166985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20102040 msgr2=0x7f9a201975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20102040 0x7f9a201975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.470+0000 7f9a262a4640 1 -- 172.21.15.26:0/26166985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2019ecc0 con 0x7f9a20100e60 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a262a4640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9a10007b50 tx=0x7f9a1000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a177fe640 1 -- 172.21.15.26:0/26166985 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10007c40 con 0x7f9a20100e60 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a177fe640 1 -- 172.21.15.26:0/26166985 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a10007de0 con 0x7f9a20100e60 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2006a2c0 con 0x7f9a20100e60 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a177fe640 1 -- 172.21.15.26:0/26166985 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000ac90 con 0x7f9a20100e60 2024-04-23T18:17:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.471+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2006a860 con 0x7f9a20100e60 2024-04-23T18:17:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 Processor -- start 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a177fe640 1 -- 172.21.15.26:0/26166985 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9a1000eb30 con 0x7f9a20100e60 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 -- start start 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c1b80 0x564cea4c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c2d50 0x564cea4c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564cea426c50 con 0x564cea4c3710 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564cea3c3830 con 0x564cea4c1b80 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564cea422c10 con 0x564cea4c2d50 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48744/0 (socket says 172.21.15.26:48744) 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a25aa3640 1 -- 172.21.15.26:0/3166967815 learned_addr learned my addr 172.21.15.26:0/3166967815 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c1b80 0x564cea4c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.474+0000 7f9a262a4640 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c2d50 0x564cea4c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 -- 172.21.15.26:0/3166967815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c2d50 msgr2=0x564cea4c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c2d50 0x564cea4c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 -- 172.21.15.26:0/3166967815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c1b80 msgr2=0x564cea4c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c1b80 0x564cea4c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 -- 172.21.15.26:0/3166967815 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cea1392d0 con 0x564cea4c3710 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c032ba0 tx=0x7f9a1c032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6835794a7c87e22c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a177fe640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 0x7f9a0c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a14bfe640 1 -- 172.21.15.26:0/3166967815 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c03f070 con 0x564cea4c3710 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a14bfe640 1 -- 172.21.15.26:0/3166967815 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1c0317a0 con 0x564cea4c3710 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a14bfe640 1 -- 172.21.15.26:0/3166967815 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c035070 con 0x564cea4c3710 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 0x7f9a0c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.475+0000 7f9a177fe640 1 -- 172.21.15.26:0/26166985 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9a100d1b60 con 0x7f9a20100e60 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3166967815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 msgr2=0x564cea4c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 0x7f9a0c0a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x564cea4c2c50 tx=0x7f9a18009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c032ba0 tx=0x7f9a1c032bd0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3166967815 shutdown_connections 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c3710 0x564cea4c9fa0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c2d50 0x564cea4c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3166967815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c1b80 0x564cea4c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3166967815 >> 172.21.15.26:0/3166967815 conn(0x564cea4bd070 msgr2=0x564cea4bd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3166967815 shutdown_connections 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3166967815 wait complete. 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 -- start start 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.476+0000 7f9a2b21d740 1 Processor -- start 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564cea4ca6c0 con 0x564cea4c2d50 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564cea3c1470 con 0x564cea4c3710 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564cea4ca840 con 0x564cea4c1b80 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a2b21d740 10 client.0 _ll_register_callbacks cb 0x564cea3e6f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a262a4640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48758/0 (socket says 172.21.15.26:48758) 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 -- 172.21.15.26:0/4129002981 learned_addr learned my addr 172.21.15.26:0/4129002981 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 -- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 msgr2=0x564cea4cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 -- 172.21.15.26:0/4129002981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 msgr2=0x564cea4d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 -- 172.21.15.26:0/4129002981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cea2036b0 con 0x564cea4c2d50 2024-04-23T18:17:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c005c40 tx=0x7f9a1c02aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb5eeeea3dd7b65 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.477+0000 7f9a09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c03f050 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9a1c004250 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a1c0317a0 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a1c035040 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 10 client.15231 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564cea4d6810 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1c0317a0 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a1c003680 con 0x564cea4c2d50 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 10 client.15231 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 10 client.15231 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 20 client.15231 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea4d8980 0x564cea4d9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 4 client.15231 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.478+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cea434670 con 0x564cea4d8980 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.479+0000 7f9a262a4640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea4d8980 0x564cea4d9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.479+0000 7f9a262a4640 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea4d8980 0x564cea4d9df0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45c862b8b6724102 server_cookie=ee11eb60fca47360 in_seq=0 out_seq=0 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.479+0000 7f9a09ffb640 10 client.15231 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a09ffb640 1 -- 172.21.15.26:0/4129002981 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564cea434670 con 0x564cea4d8980 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a09ffb640 10 client.15231 handle_command_reply: tid=0 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 client.15231 shutdown 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 msgr2=0x564cea4cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c005c40 tx=0x7f9a1c02aa90 comp rx=0 tx=0).stop 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 shutdown_connections 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea4d8980 0x564cea4d9df0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4cc460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a09ffb640 0 client.15231 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/4129002981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 >> 172.21.15.26:0/4129002981 conn(0x564cea4bd070 msgr2=0x564cea4bc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 shutdown_connections 2024-04-23T18:17:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 1 -- 172.21.15.26:0/4129002981 wait complete. 2024-04-23T18:17:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.480+0000 7f9a2b21d740 20 client.15231 trim_cache size 0 max 16384 2024-04-23T18:17:00.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:00 smithi131 ceph-mon[33041]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:00.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:00 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1406138722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:00 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3467666128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:00.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:00 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4207019977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 Processor -- start 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 -- start start 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c2d50 0x564cea4b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c3710 0x564cea4bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564cea4d8130 con 0x564cea4c1b80 2024-04-23T18:17:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564cea4d8800 con 0x564cea4c2d50 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564cea4d8ed0 con 0x564cea4c3710 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a262a4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a262a4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48770/0 (socket says 172.21.15.26:48770) 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.509+0000 7f9a262a4640 1 -- 172.21.15.26:0/3202110976 learned_addr learned my addr 172.21.15.26:0/3202110976 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c2d50 0x564cea4b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c3710 0x564cea4bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 -- 172.21.15.26:0/3202110976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c3710 msgr2=0x564cea4bfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c3710 0x564cea4bfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 -- 172.21.15.26:0/3202110976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c2d50 msgr2=0x564cea4b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c2d50 0x564cea4b80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 -- 172.21.15.26:0/3202110976 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cea4c9a20 con 0x564cea4c1b80 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a262a4640 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9a1009a8b0 tx=0x7f9a1000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4684f6766d4585 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a0bfff640 1 -- 172.21.15.26:0/3202110976 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1009e5e0 con 0x564cea4c1b80 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a0bfff640 1 -- 172.21.15.26:0/3202110976 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1009dee0 con 0x564cea4c1b80 2024-04-23T18:17:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.510+0000 7f9a0bfff640 1 -- 172.21.15.26:0/3202110976 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10013dd0 con 0x564cea4c1b80 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3202110976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 msgr2=0x564cea4c3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9a1009a8b0 tx=0x7f9a1000a730 comp rx=0 tx=0).stop 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3202110976 shutdown_connections 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c2d50 0x564cea4b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c3710 0x564cea4bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/3202110976 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c1b80 0x564cea4c3130 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3202110976 >> 172.21.15.26:0/3202110976 conn(0x564cea4bd070 msgr2=0x564cea4bd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3202110976 shutdown_connections 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- 172.21.15.26:0/3202110976 wait complete. 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 -- start start 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.511+0000 7f9a2b21d740 1 Processor -- start 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564cea4d9c70 con 0x564cea4c2d50 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564cea4d95a0 con 0x564cea4c3710 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564cea4da340 con 0x564cea4c1b80 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 10 client.0 _ll_register_callbacks cb 0x7f9a180098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48782/0 (socket says 172.21.15.26:48782) 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 -- 172.21.15.26:0/381174162 learned_addr learned my addr 172.21.15.26:0/381174162 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a26aa5640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a262a4640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 -- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 msgr2=0x564cea4ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 -- 172.21.15.26:0/381174162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 msgr2=0x564cea4d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 -- 172.21.15.26:0/381174162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cea4d1de0 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a25aa3640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c003a20 tx=0x7f9a1c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebce93b95f526423 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c05c030 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9a1c004460 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a1009dee0 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.512+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a1c03f070 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 10 client.15237 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564cea53ead0 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1009dee0 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a1c0321c0 con 0x564cea4c2d50 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 10 client.15237 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 20 client.15237 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea5436b0 0x564cea5423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 4 client.15237 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "status"}) -- 0x564cea42e4e0 con 0x564cea5436b0 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.513+0000 7f9a262a4640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea5436b0 0x564cea5423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a09ffb640 10 client.15237 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a262a4640 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea5436b0 0x564cea5423c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db00cacdb6057aae server_cookie=dae526df0924ca7b in_seq=0 out_seq=0 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a09ffb640 1 -- 172.21.15.26:0/381174162 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x564cea42e4e0 con 0x564cea5436b0 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a09ffb640 10 client.15237 handle_command_reply: tid=0 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 client.15237 shutdown 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 msgr2=0x564cea4ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c003a20 tx=0x7f9a1c003820 comp rx=0 tx=0).stop 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 shutdown_connections 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x564cea5436b0 0x564cea5423c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564cea4c3710 0x564cea4d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564cea4c2d50 0x564cea4ce7d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a09ffb640 0 client.15237 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 --2- 172.21.15.26:0/381174162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cea4c1b80 0x564cea4ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 >> 172.21.15.26:0/381174162 conn(0x564cea4bd070 msgr2=0x564cea4b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 shutdown_connections 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 1 -- 172.21.15.26:0/381174162 wait complete. 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.514+0000 7f9a2b21d740 20 client.15237 trim_cache size 0 max 16384 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 2, 2024-04-23T18:17:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24460, 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 109.815062422, 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 128.412296384 2024-04-23T18:17:00.515 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 msgr2=0x7f9a0c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 0x7f9a0c0a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x564cea4c2c50 tx=0x7f9a18009200 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 msgr2=0x7f9a201970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9a10007b50 tx=0x7f9a1000e420 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 shutdown_connections 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9a0c0a3c30 0x7f9a0c0a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a20102040 0x7f9a201975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a20102a00 0x7f9a2019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 --2- 172.21.15.26:0/26166985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9a20100e60 0x7f9a201970a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 >> 172.21.15.26:0/26166985 conn(0x7f9a200fc440 msgr2=0x7f9a20105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 shutdown_connections 2024-04-23T18:17:00.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.515+0000 7f9a28508640 1 -- 172.21.15.26:0/26166985 wait complete. 2024-04-23T18:17:00.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 -- 172.21.15.26:0/550651758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358102ad0 msgr2=0x7fb358104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 --2- 172.21.15.26:0/550651758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358102ad0 0x7fb358104ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb354009900 tx=0x7fb35402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 -- 172.21.15.26:0/550651758 shutdown_connections 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 --2- 172.21.15.26:0/550651758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358107d30 0x7fb35810a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 --2- 172.21.15.26:0/550651758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 0x7fb3581077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 --2- 172.21.15.26:0/550651758 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358102ad0 0x7fb358104ec0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 -- 172.21.15.26:0/550651758 >> 172.21.15.26:0/550651758 conn(0x7fb3580fc400 msgr2=0x7fb3580fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 -- 172.21.15.26:0/550651758 shutdown_connections 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.631+0000 7fb360605640 1 -- 172.21.15.26:0/550651758 wait complete. 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 Processor -- start 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 -- start start 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358102ad0 0x7fb358198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 0x7fb358198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810dae0 con 0x7fb358107d30 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810d960 con 0x7fb358102ad0 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb360605640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810dc60 con 0x7fb358105400 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48814/0 (socket says 172.21.15.26:48814) 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 -- 172.21.15.26:0/1046295981 learned_addr learned my addr 172.21.15.26:0/1046295981 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358102ad0 0x7fb358198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35dba0640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 0x7fb358198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 -- 172.21.15.26:0/1046295981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 msgr2=0x7fb358198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 0x7fb358198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 -- 172.21.15.26:0/1046295981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358102ad0 msgr2=0x7fb358198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358102ad0 0x7fb358198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.632+0000 7fb35eba2640 1 -- 172.21.15.26:0/1046295981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35819fe40 con 0x7fb358107d30 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb35eba2640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb35000eaa0 tx=0x7fb35000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb34f7fe640 1 -- 172.21.15.26:0/1046295981 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35000cdb0 con 0x7fb358107d30 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3581a0130 con 0x7fb358107d30 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb34f7fe640 1 -- 172.21.15.26:0/1046295981 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb350005020 con 0x7fb358107d30 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb34f7fe640 1 -- 172.21.15.26:0/1046295981 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500116d0 con 0x7fb358107d30 2024-04-23T18:17:00.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.633+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3581a0680 con 0x7fb358107d30 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 Processor -- start 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb34f7fe640 1 -- 172.21.15.26:0/1046295981 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb3500051d0 con 0x7fb358107d30 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 -- start start 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x55817662b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662b710 0x558176631fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55817658ec50 con 0x558176629b80 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55817652b830 con 0x55817662ad50 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.636+0000 7fb3633f8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55817658ac10 con 0x55817662b710 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48830/0 (socket says 172.21.15.26:48830) 2024-04-23T18:17:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 -- 172.21.15.26:0/3260060776 learned_addr learned my addr 172.21.15.26:0/3260060776 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x55817662b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662b710 0x558176631fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 -- 172.21.15.26:0/3260060776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662b710 msgr2=0x558176631fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662b710 0x558176631fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb34f7fe640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 0x7fb3440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 -- 172.21.15.26:0/3260060776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 msgr2=0x55817662b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x55817662b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 -- 172.21.15.26:0/3260060776 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581762a12d0 con 0x558176629b80 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35eba2640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 0x7fb3440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb35dba0640 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb3480135b0 tx=0x7fb348009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b98009742401d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb34f7fe640 1 -- 172.21.15.26:0/1046295981 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb3500d05b0 con 0x7fb358107d30 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb34cbfe640 1 -- 172.21.15.26:0/3260060776 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004050 con 0x558176629b80 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb34cbfe640 1 -- 172.21.15.26:0/3260060776 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3480054d0 con 0x558176629b80 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.637+0000 7fb34cbfe640 1 -- 172.21.15.26:0/3260060776 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004bc0 con 0x558176629b80 2024-04-23T18:17:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 -- 172.21.15.26:0/3260060776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 msgr2=0x558176629fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb35eba2640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 0x7fb3440a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb35000efd0 tx=0x7fb35001c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb3480135b0 tx=0x7fb348009830 comp rx=0 tx=0).stop 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 -- 172.21.15.26:0/3260060776 shutdown_connections 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x55817662b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662b710 0x558176631fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 --2- 172.21.15.26:0/3260060776 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176629fa0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 -- 172.21.15.26:0/3260060776 >> 172.21.15.26:0/3260060776 conn(0x558176625070 msgr2=0x558176625470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 -- 172.21.15.26:0/3260060776 shutdown_connections 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.638+0000 7fb3633f8740 1 -- 172.21.15.26:0/3260060776 wait complete. 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 -- start start 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 Processor -- start 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558176629b80 0x558176633fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x5581766344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558176632720 con 0x55817662b710 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558176529470 con 0x55817662ad50 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5581766328a0 con 0x558176629b80 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb3633f8740 10 client.0 _ll_register_callbacks cb 0x55817654ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48832/0 (socket says 172.21.15.26:48832) 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 -- 172.21.15.26:0/1494481040 learned_addr learned my addr 172.21.15.26:0/1494481040 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558176629b80 0x558176633fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x5581766344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 -- 172.21.15.26:0/1494481040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558176629b80 msgr2=0x558176633fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558176629b80 0x558176633fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 -- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 msgr2=0x5581766344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x5581766344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.639+0000 7fb35dba0640 1 -- 172.21.15.26:0/1494481040 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55817636b6b0 con 0x55817662b710 2024-04-23T18:17:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb35dba0640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb348005fd0 tx=0x7fb348009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f80c4b25fe8b97 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004dd0 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb34800f580 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3480054d0 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb34800f780 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 10 client.15249 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55817663e8a0 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3480054d0 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb34801d020 con 0x55817662b710 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 10 client.15249 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 10 client.15249 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 20 client.15249 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x558176640c60 0x5581766420d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 4 client.15249 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.640+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55817659c670 con 0x558176640c60 2024-04-23T18:17:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.641+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x558176640c60 0x5581766420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.641+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x558176640c60 0x5581766420d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=879f218f222e36fe server_cookie=9017006aebdf3f46 in_seq=0 out_seq=0 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.641+0000 7fb341ffb640 10 client.15249 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb341ffb640 1 -- 172.21.15.26:0/1494481040 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55817659c670 con 0x558176640c60 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb341ffb640 10 client.15249 handle_command_reply: tid=0 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 client.15249 shutdown 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 msgr2=0x55817663b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb348005fd0 tx=0x7fb348009640 comp rx=0 tx=0).stop 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 shutdown_connections 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x558176640c60 0x5581766420d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662ad50 0x5581766344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b530 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb341ffb640 0 client.15249 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 --2- 172.21.15.26:0/1494481040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558176629b80 0x558176633fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 >> 172.21.15.26:0/1494481040 conn(0x558176625070 msgr2=0x5581766241d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 shutdown_connections 2024-04-23T18:17:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 1 -- 172.21.15.26:0/1494481040 wait complete. 2024-04-23T18:17:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.642+0000 7fb3633f8740 20 client.15249 trim_cache size 0 max 16384 2024-04-23T18:17:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.661+0000 7fb3633f8740 1 Processor -- start 2024-04-23T18:17:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.661+0000 7fb3633f8740 1 -- start start 2024-04-23T18:17:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558176629b80 0x558176632c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176634450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558176640410 con 0x55817662b710 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558176640ae0 con 0x558176629b80 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb3633f8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5581766411b0 con 0x55817662ad50 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48834/0 (socket says 172.21.15.26:48834) 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 -- 172.21.15.26:0/257035496 learned_addr learned my addr 172.21.15.26:0/257035496 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176634450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558176629b80 0x558176632c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 -- 172.21.15.26:0/257035496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 msgr2=0x558176634450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176634450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 -- 172.21.15.26:0/257035496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558176629b80 msgr2=0x558176632c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558176629b80 0x558176632c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 -- 172.21.15.26:0/257035496 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55817662a7d0 con 0x55817662b710 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176634450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.662+0000 7fb35dba0640 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb348009880 tx=0x7fb348009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77a355d52a5b30b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb343fff640 1 -- 172.21.15.26:0/257035496 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348005420 con 0x55817662b710 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb343fff640 1 -- 172.21.15.26:0/257035496 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb348025030 con 0x55817662b710 2024-04-23T18:17:00.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb343fff640 1 -- 172.21.15.26:0/257035496 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34801f450 con 0x55817662b710 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 -- 172.21.15.26:0/257035496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 msgr2=0x55817663b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb348009880 tx=0x7fb348009f30 comp rx=0 tx=0).stop 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 -- 172.21.15.26:0/257035496 shutdown_connections 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55817662b710 0x55817663b450 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176634450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 --2- 172.21.15.26:0/257035496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558176629b80 0x558176632c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 -- 172.21.15.26:0/257035496 >> 172.21.15.26:0/257035496 conn(0x558176625070 msgr2=0x558176631e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 -- 172.21.15.26:0/257035496 shutdown_connections 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.663+0000 7fb3633f8740 1 -- 172.21.15.26:0/257035496 wait complete. 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 -- start start 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 Processor -- start 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176636eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662b710 0x558176639ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558176641f50 con 0x558176629b80 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558176641880 con 0x55817662b710 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x558176642620 con 0x55817662ad50 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb3633f8740 10 client.0 _ll_register_callbacks cb 0x7fb354009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48840/0 (socket says 172.21.15.26:48840) 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 -- 172.21.15.26:0/557120675 learned_addr learned my addr 172.21.15.26:0/557120675 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662b710 0x558176639ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 -- 172.21.15.26:0/557120675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 msgr2=0x558176636eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176636eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 -- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662b710 msgr2=0x558176639ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662b710 0x558176639ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.664+0000 7fb35dba0640 1 -- 172.21.15.26:0/557120675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55817663a4c0 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb35dba0640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6cdba47fdf1919 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348005c10 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb348005420 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb348025030 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb348005620 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 10 client.15255 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55817663b7c0 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb348025030 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb348043020 con 0x558176629b80 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 10 client.15255 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 10 client.15255 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 20 client.15255 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5581766ab440 0x5581766aa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 4 client.15255 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.665+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "status"}) -- 0x5581765964e0 con 0x5581766ab440 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5581766ab440 0x5581766aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb35e3a1640 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5581766ab440 0x5581766aa150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36a5590168583a49 server_cookie=c3a71f6d93677755 in_seq=0 out_seq=0 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb341ffb640 10 client.15255 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb341ffb640 1 -- 172.21.15.26:0/557120675 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5581765964e0 con 0x5581766ab440 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb341ffb640 10 client.15255 handle_command_reply: tid=0 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 client.15255 shutdown 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 msgr2=0x558176636970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348005fb0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 shutdown_connections 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x5581766ab440 0x5581766aa150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55817662b710 0x558176639ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55817662ad50 0x558176636eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 --2- 172.21.15.26:0/557120675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558176629b80 0x558176636970 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 >> 172.21.15.26:0/557120675 conn(0x558176625070 msgr2=0x5581766200b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.666+0000 7fb341ffb640 0 client.15255 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 shutdown_connections 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb3633f8740 1 -- 172.21.15.26:0/557120675 wait complete. 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb3633f8740 20 client.15255 trim_cache size 0 max 16384 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 1, 2024-04-23T18:17:00.666 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24464, 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 110.97316363100001, 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 131.26719034199999 2024-04-23T18:17:00.667 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 msgr2=0x7fb3440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 0x7fb3440a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb35000efd0 tx=0x7fb35001c040 comp rx=0 tx=0).stop 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 msgr2=0x7fb35819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb35000eaa0 tx=0x7fb35000ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 shutdown_connections 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb3440a3bb0 0x7fb3440a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb358107d30 0x7fb35819f7d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb358105400 0x7fb358198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 --2- 172.21.15.26:0/1046295981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb358102ad0 0x7fb358198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 >> 172.21.15.26:0/1046295981 conn(0x7fb3580fc400 msgr2=0x7fb358103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 shutdown_connections 2024-04-23T18:17:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.667+0000 7fb360605640 1 -- 172.21.15.26:0/1046295981 wait complete. 2024-04-23T18:17:00.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:17:00.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 -- 172.21.15.26:0/3749856432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 msgr2=0x7faa10104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 --2- 172.21.15.26:0/3749856432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10104e00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009900 tx=0x7faa0c02a410 comp rx=0 tx=0).stop 2024-04-23T18:17:00.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 -- 172.21.15.26:0/3749856432 shutdown_connections 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 --2- 172.21.15.26:0/3749856432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10107c70 0x7faa1010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 --2- 172.21.15.26:0/3749856432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10105340 0x7faa10107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 --2- 172.21.15.26:0/3749856432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10104e00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.784+0000 7faa180b4640 1 -- 172.21.15.26:0/3749856432 >> 172.21.15.26:0/3749856432 conn(0x7faa100fc400 msgr2=0x7faa100fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- 172.21.15.26:0/3749856432 shutdown_connections 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- 172.21.15.26:0/3749856432 wait complete. 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 Processor -- start 2024-04-23T18:17:00.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- start start 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 0x7faa101009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 0x7faa10100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010dad0 con 0x7faa10102a10 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d950 con 0x7faa10105340 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa180b4640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010dc50 con 0x7faa10107c70 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa15e50640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa15e50640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48852/0 (socket says 172.21.15.26:48852) 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.785+0000 7faa15e50640 1 -- 172.21.15.26:0/423570655 learned_addr learned my addr 172.21.15.26:0/423570655 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa1564f640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 0x7faa101009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa16651640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 0x7faa10100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 -- 172.21.15.26:0/423570655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 msgr2=0x7faa10100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 0x7faa10100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 -- 172.21.15.26:0/423570655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 msgr2=0x7faa101009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 0x7faa101009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 -- 172.21.15.26:0/423570655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa101017c0 con 0x7faa10102a10 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa16651640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 0x7faa10100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa1564f640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 0x7faa101009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa15e50640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005940 tx=0x7faa0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa06ffd640 1 -- 172.21.15.26:0/423570655 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c03f030 con 0x7faa10102a10 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa101a2330 con 0x7faa10102a10 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa06ffd640 1 -- 172.21.15.26:0/423570655 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c003cb0 con 0x7faa10102a10 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa06ffd640 1 -- 172.21.15.26:0/423570655 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c035070 con 0x7faa10102a10 2024-04-23T18:17:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.786+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa101a27f0 con 0x7faa10102a10 2024-04-23T18:17:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.789+0000 7faa06ffd640 1 -- 172.21.15.26:0/423570655 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faa0c042030 con 0x7faa10102a10 2024-04-23T18:17:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa06ffd640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 0x7fa9fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 Processor -- start 2024-04-23T18:17:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1564f640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 0x7fa9fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa06ffd640 1 -- 172.21.15.26:0/423570655 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faa0c0f0960 con 0x7faa10102a10 2024-04-23T18:17:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 -- start start 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d281fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d289fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d1e6c50 con 0x55ad8d282d50 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d183830 con 0x55ad8d281b80 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.790+0000 7faa1adc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d1e2c10 con 0x55ad8d283710 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48858/0 (socket says 172.21.15.26:48858) 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 -- 172.21.15.26:0/2211499937 learned_addr learned my addr 172.21.15.26:0/2211499937 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 0x7fa9fc0a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa101020a0 tx=0x7faa00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa15e50640 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d281fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa16651640 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d289fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 -- 172.21.15.26:0/2211499937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 msgr2=0x55ad8d289fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d289fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 -- 172.21.15.26:0/2211499937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 msgr2=0x55ad8d281fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d281fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 -- 172.21.15.26:0/2211499937 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8cef92d0 con 0x55ad8d282d50 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7faa1564f640 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faa00012010 tx=0x7faa00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e482930edf02aa38 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7fa9fbfff640 1 -- 172.21.15.26:0/2211499937 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0001f620 con 0x55ad8d282d50 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7fa9fbfff640 1 -- 172.21.15.26:0/2211499937 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0001fc70 con 0x55ad8d282d50 2024-04-23T18:17:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.791+0000 7fa9fbfff640 1 -- 172.21.15.26:0/2211499937 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00027c70 con 0x55ad8d282d50 2024-04-23T18:17:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 -- 172.21.15.26:0/2211499937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 msgr2=0x55ad8d2831d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faa00012010 tx=0x7faa00014000 comp rx=0 tx=0).stop 2024-04-23T18:17:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 -- 172.21.15.26:0/2211499937 shutdown_connections 2024-04-23T18:17:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d289fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d2831d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 --2- 172.21.15.26:0/2211499937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d281fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 -- 172.21.15.26:0/2211499937 >> 172.21.15.26:0/2211499937 conn(0x55ad8d27d070 msgr2=0x55ad8d27d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 -- 172.21.15.26:0/2211499937 shutdown_connections 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.792+0000 7faa1adc5740 1 -- 172.21.15.26:0/2211499937 wait complete. 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 -- start start 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 Processor -- start 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d281b80 0x55ad8d28c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d293570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d28a720 con 0x55ad8d282d50 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d181470 con 0x55ad8d283710 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.793+0000 7faa1adc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d28a8a0 con 0x55ad8d281b80 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1adc5740 10 client.0 _ll_register_callbacks cb 0x55ad8d1a6f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48864/0 (socket says 172.21.15.26:48864) 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 -- 172.21.15.26:0/208522498 learned_addr learned my addr 172.21.15.26:0/208522498 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa15e50640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d281b80 0x55ad8d28c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa16651640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d293570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 -- 172.21.15.26:0/208522498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d281b80 msgr2=0x55ad8d28c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d281b80 0x55ad8d28c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 -- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 msgr2=0x55ad8d293570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d293570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 -- 172.21.15.26:0/208522498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8cfc36b0 con 0x55ad8d282d50 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1564f640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faa000096f0 tx=0x7faa00028c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc2bc7c8cbf85cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7fa9f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00037060 con 0x55ad8d282d50 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faa00011c80 con 0x55ad8d282d50 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0001fc70 con 0x55ad8d282d50 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa00028e20 con 0x55ad8d282d50 2024-04-23T18:17:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1adc5740 10 client.15267 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.794+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ad8d2968e0 con 0x55ad8d282d50 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0001fc70 con 0x55ad8d282d50 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7faa000046c0 con 0x55ad8d282d50 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 10 client.15267 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 10 client.15267 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 20 client.15267 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d298b50 0x55ad8d299fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 4 client.15267 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad8d1f4670 con 0x55ad8d298b50 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.795+0000 7faa15e50640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d298b50 0x55ad8d299fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7faa15e50640 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d298b50 0x55ad8d299fc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9171ae433788f3 server_cookie=26d466c7e9feb8f1 in_seq=0 out_seq=0 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7fa9f97fa640 10 client.15267 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7fa9f97fa640 1 -- 172.21.15.26:0/208522498 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad8d1f4670 con 0x55ad8d298b50 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7fa9f97fa640 10 client.15267 handle_command_reply: tid=0 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7faa1adc5740 1 client.15267 shutdown 2024-04-23T18:17:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 msgr2=0x55ad8d28c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.796+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faa000096f0 tx=0x7faa00028c30 comp rx=0 tx=0).stop 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 shutdown_connections 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d298b50 0x55ad8d299fc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d293570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d28c5a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 --2- 172.21.15.26:0/208522498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d281b80 0x55ad8d28c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7fa9f97fa640 0 client.15267 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 >> 172.21.15.26:0/208522498 conn(0x55ad8d27d070 msgr2=0x55ad8d27c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 shutdown_connections 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 1 -- 172.21.15.26:0/208522498 wait complete. 2024-04-23T18:17:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.797+0000 7faa1adc5740 20 client.15267 trim_cache size 0 max 16384 2024-04-23T18:17:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 Processor -- start 2024-04-23T18:17:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 -- start start 2024-04-23T18:17:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d282d50 0x55ad8d2780b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d27b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d298300 con 0x55ad8d281b80 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d2989d0 con 0x55ad8d283710 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.815+0000 7faa1adc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d2990a0 con 0x55ad8d282d50 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48876/0 (socket says 172.21.15.26:48876) 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 -- 172.21.15.26:0/1043151509 learned_addr learned my addr 172.21.15.26:0/1043151509 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa1564f640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d282d50 0x55ad8d2780b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa16651640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d27b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 -- 172.21.15.26:0/1043151509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d282d50 msgr2=0x55ad8d2780b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d282d50 0x55ad8d2780b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 -- 172.21.15.26:0/1043151509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 msgr2=0x55ad8d27b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d27b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 -- 172.21.15.26:0/1043151509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8d2827d0 con 0x55ad8d281b80 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa16651640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d27b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:00.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7faa15e50640 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0bcb80 tx=0x7faa0c0dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1995d9bf13be4bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7fa9fb7fe640 1 -- 172.21.15.26:0/1043151509 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c049040 con 0x55ad8d281b80 2024-04-23T18:17:00.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7fa9fb7fe640 1 -- 172.21.15.26:0/1043151509 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c0a7040 con 0x55ad8d281b80 2024-04-23T18:17:00.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.816+0000 7fa9fb7fe640 1 -- 172.21.15.26:0/1043151509 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0baef0 con 0x55ad8d281b80 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 -- 172.21.15.26:0/1043151509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 msgr2=0x55ad8d2998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0bcb80 tx=0x7faa0c0dd040 comp rx=0 tx=0).stop 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 -- 172.21.15.26:0/1043151509 shutdown_connections 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d283710 0x55ad8d27b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d282d50 0x55ad8d2780b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1043151509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d281b80 0x55ad8d2998f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 -- 172.21.15.26:0/1043151509 >> 172.21.15.26:0/1043151509 conn(0x55ad8d27d070 msgr2=0x55ad8d285d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 -- 172.21.15.26:0/1043151509 shutdown_connections 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.817+0000 7faa1adc5740 1 -- 172.21.15.26:0/1043151509 wait complete. 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 -- start start 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 Processor -- start 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d284900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d285380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d299e40 con 0x55ad8d282d50 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d299770 con 0x55ad8d281b80 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ad8d29a510 con 0x55ad8d283710 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1adc5740 10 client.0 _ll_register_callbacks cb 0x7faa0c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1564f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48884/0 (socket says 172.21.15.26:48884) 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa1564f640 1 -- 172.21.15.26:0/1122163652 learned_addr learned my addr 172.21.15.26:0/1122163652 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.818+0000 7faa16651640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d285380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa15e50640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d284900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 -- 172.21.15.26:0/1122163652 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 msgr2=0x55ad8d285380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d285380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 -- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 msgr2=0x55ad8d284900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d284900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 -- 172.21.15.26:0/1122163652 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8d2917c0 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa16651640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d285380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1564f640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faa0003d470 tx=0x7faa00005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f7f307330c8ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00042020 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faa00004660 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0c0a7040 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa00037040 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1adc5740 10 client.15273 fetch_fsmap learned FSMap version 21 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ad8d292a30 con 0x55ad8d282d50 2024-04-23T18:17:00.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c0a7040 con 0x55ad8d282d50 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7faa00027c40 con 0x55ad8d282d50 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1adc5740 10 client.15273 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.819+0000 7faa1adc5740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa1adc5740 20 client.15273 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d3030a0 0x55ad8d2ff4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa1adc5740 4 client.15273 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "status"}) -- 0x55ad8d1ee4e0 con 0x55ad8d3030a0 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa15e50640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d3030a0 0x55ad8d2ff4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7faa15e50640 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d3030a0 0x55ad8d2ff4e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=119e7add30aeee9b server_cookie=d620318b5eac20f4 in_seq=0 out_seq=0 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.820+0000 7fa9f97fa640 10 client.15273 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7fa9f97fa640 1 -- 172.21.15.26:0/1122163652 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ad8d1ee4e0 con 0x55ad8d3030a0 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7fa9f97fa640 10 client.15273 handle_command_reply: tid=0 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 client.15273 shutdown 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 msgr2=0x55ad8d284e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faa0003d470 tx=0x7faa00005930 comp rx=0 tx=0).stop 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 shutdown_connections 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ad8d3030a0 0x55ad8d2ff4e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ad8d283710 0x55ad8d285380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ad8d282d50 0x55ad8d284e40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 --2- 172.21.15.26:0/1122163652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad8d281b80 0x55ad8d284900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7fa9f97fa640 0 client.15273 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 >> 172.21.15.26:0/1122163652 conn(0x55ad8d27d070 msgr2=0x55ad8d27ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 shutdown_connections 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 1 -- 172.21.15.26:0/1122163652 wait complete. 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.821+0000 7faa1adc5740 20 client.15273 trim_cache size 0 max 16384 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:00.820 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 0, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24503, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 114.14695645499999, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 21, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 88, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 86, 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 120.160257774 2024-04-23T18:17:00.821 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 msgr2=0x7fa9fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 0x7fa9fc0a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa101020a0 tx=0x7faa00005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 msgr2=0x7faa10102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005940 tx=0x7faa0c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 shutdown_connections 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fa9fc0a3bb0 0x7fa9fc0a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa10105340 0x7faa101009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa10107c70 0x7faa10100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 --2- 172.21.15.26:0/423570655 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa10102a10 0x7faa10102330 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 >> 172.21.15.26:0/423570655 conn(0x7faa100fc400 msgr2=0x7faa10103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 shutdown_connections 2024-04-23T18:17:00.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.822+0000 7faa180b4640 1 -- 172.21.15.26:0/423570655 wait complete. 2024-04-23T18:17:00.826 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T18:17:00.836 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T18:17:00.836 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T18:17:00.836 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T18:17:00.836 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-23T18:17:00.837 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:17:00.837 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:17:00.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-23T18:17:00.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- 172.21.15.26:0/3535875311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 msgr2=0x7f784c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 --2- 172.21.15.26:0/3535875311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c109320 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7844009ca0 tx=0x7f784402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- 172.21.15.26:0/3535875311 shutdown_connections 2024-04-23T18:17:00.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 --2- 172.21.15.26:0/3535875311 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c109320 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 --2- 172.21.15.26:0/3535875311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c102040 0x7f784c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 --2- 172.21.15.26:0/3535875311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c100e60 0x7f784c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- 172.21.15.26:0/3535875311 >> 172.21.15.26:0/3535875311 conn(0x7f784c0fc420 msgr2=0x7f784c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- 172.21.15.26:0/3535875311 shutdown_connections 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- 172.21.15.26:0/3535875311 wait complete. 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 Processor -- start 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.947+0000 7f785199b640 1 -- start start 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c100e60 0x7f784c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 0x7f784c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f784c10ca00 con 0x7f784c102a00 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f784c10c880 con 0x7f784c102040 2024-04-23T18:17:00.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f784c10cb80 con 0x7f784c100e60 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48898/0 (socket says 172.21.15.26:48898) 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 -- 172.21.15.26:0/2035153732 learned_addr learned my addr 172.21.15.26:0/2035153732 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784affd640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c100e60 0x7f784c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784a7fc640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 0x7f784c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 -- 172.21.15.26:0/2035153732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c100e60 msgr2=0x7f784c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c100e60 0x7f784c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 -- 172.21.15.26:0/2035153732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 msgr2=0x7f784c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 0x7f784c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 -- 172.21.15.26:0/2035153732 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784c19fc70 con 0x7f784c102a00 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784a7fc640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 0x7f784c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f784b7fe640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f78440097b0 tx=0x7f7844031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7844003bf0 con 0x7f784c102a00 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7844003d90 con 0x7f784c102a00 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f784c19ff00 con 0x7f784c102a00 2024-04-23T18:17:00.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784402fc70 con 0x7f784c102a00 2024-04-23T18:17:00.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.948+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f784c1a03c0 con 0x7f784c102a00 2024-04-23T18:17:00.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.950+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f784c1a0620 con 0x7f784c102a00 2024-04-23T18:17:00.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.951+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7844004040 con 0x7f784c102a00 2024-04-23T18:17:00.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.951+0000 7f7850999640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 0x7f78340a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:00.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.951+0000 7f784affd640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 0x7f78340a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:00.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.951+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f78440f1a30 con 0x7f784c102a00 2024-04-23T18:17:00.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.952+0000 7f784affd640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 0x7f78340a6110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f784c101bb0 tx=0x7f7840005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:00.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:00.956+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78440be420 con 0x7f784c102a00 2024-04-23T18:17:01.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:01.219+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f784c1a07f0 con 0x7f784c102a00 2024-04-23T18:17:01.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:01 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4031640523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:01.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:01 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4031640523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:01.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:01 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4031640523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:02.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.074+0000 7f7850999640 1 -- 172.21.15.26:0/2035153732 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f78440bdd20 con 0x7f784c102a00 2024-04-23T18:17:02.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.076+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 msgr2=0x7f78340a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.076+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 0x7f78340a6110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f784c101bb0 tx=0x7f7840005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:02.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.076+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 msgr2=0x7f784c19f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.076+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f78440097b0 tx=0x7f7844031840 comp rx=0 tx=0).stop 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 shutdown_connections 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f78340a3c50 0x7f78340a6110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f784c102040 0x7f784c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f784c102a00 0x7f784c19f600 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 --2- 172.21.15.26:0/2035153732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f784c100e60 0x7f784c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 >> 172.21.15.26:0/2035153732 conn(0x7f784c0fc420 msgr2=0x7f784c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 shutdown_connections 2024-04-23T18:17:02.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.077+0000 7f785199b640 1 -- 172.21.15.26:0/2035153732 wait complete. 2024-04-23T18:17:02.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 -- 172.21.15.26:0/3258628973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7741008a0 msgr2=0x7fb774100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 --2- 172.21.15.26:0/3258628973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7741008a0 0x7fb774100cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb764009900 tx=0x7fb76402a480 comp rx=0 tx=0).stop 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 -- 172.21.15.26:0/3258628973 shutdown_connections 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 --2- 172.21.15.26:0/3258628973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101ba0 0x7fb774102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 --2- 172.21.15.26:0/3258628973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb774101200 0x7fb774101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 --2- 172.21.15.26:0/3258628973 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb7741008a0 0x7fb774100cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 -- 172.21.15.26:0/3258628973 >> 172.21.15.26:0/3258628973 conn(0x7fb7740fc420 msgr2=0x7fb7740fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 -- 172.21.15.26:0/3258628973 shutdown_connections 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.205+0000 7fb77be89640 1 -- 172.21.15.26:0/3258628973 wait complete. 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 Processor -- start 2024-04-23T18:17:02.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 -- start start 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7741008a0 0x7fb77419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101200 0x7fb77419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410fcf0 con 0x7fb774101ba0 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410fb70 con 0x7fb774101200 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77be89640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410fe70 con 0x7fb7741008a0 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48946/0 (socket says 172.21.15.26:48946) 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 -- 172.21.15.26:0/1173652614 learned_addr learned my addr 172.21.15.26:0/1173652614 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb779424640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101200 0x7fb77419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb779c25640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7741008a0 0x7fb77419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7741008a0 msgr2=0x7fb77419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7741008a0 0x7fb77419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101200 msgr2=0x7fb77419a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101200 0x7fb77419a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.206+0000 7fb77a426640 1 -- 172.21.15.26:0/1173652614 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7741a1f60 con 0x7fb774101ba0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb77a426640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00eb10 tx=0x7fb76c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c004b20 con 0x7fb774101ba0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7741a2250 con 0x7fb774101ba0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb76c004cc0 con 0x7fb774101ba0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c010670 con 0x7fb774101ba0 2024-04-23T18:17:02.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.207+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7741a2750 con 0x7fb774101ba0 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.208+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77410b5f0 con 0x7fb774101ba0 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.209+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fb76c005020 con 0x7fb774101ba0 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.209+0000 7fb76affd640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 0x7fb7600a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.210+0000 7fb779c25640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 msgr2=0x7fb7600a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.210+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb76c0d0a60 con 0x7fb774101ba0 2024-04-23T18:17:02.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.210+0000 7fb779c25640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 0x7fb7600a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:17:02.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.214+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb76c09d3f0 con 0x7fb774101ba0 2024-04-23T18:17:02.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.407+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fb774062b50 con 0x7fb774101ba0 2024-04-23T18:17:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.411+0000 7fb779c25640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 msgr2=0x7fb7600a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.411+0000 7fb779c25640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 0x7fb7600a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:17:02.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.411+0000 7fb76affd640 1 -- 172.21.15.26:0/1173652614 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fb76c09ccf0 con 0x7fb774101ba0 2024-04-23T18:17:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 msgr2=0x7fb7600a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:17:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 0x7fb7600a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 msgr2=0x7fb7741a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00eb10 tx=0x7fb76c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 shutdown_connections 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fb7600a3ca0 0x7fb7600a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb774101200 0x7fb77419a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb774101ba0 0x7fb7741a18f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 --2- 172.21.15.26:0/1173652614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7741008a0 0x7fb77419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 >> 172.21.15.26:0/1173652614 conn(0x7fb7740fc420 msgr2=0x7fb7740ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 shutdown_connections 2024-04-23T18:17:02.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.413+0000 7fb77be89640 1 -- 172.21.15.26:0/1173652614 wait complete. 2024-04-23T18:17:02.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-mon[29364]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:02.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:17:02.418 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setuser ceph since I am not root 2024-04-23T18:17:02.418 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: ignoring --setgroup ceph since I am not root 2024-04-23T18:17:02.418 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:02.266+0000 7f7bbc404200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:17:02.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-23T18:17:02.429 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-mon[33041]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:02.429 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:17:02.430 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: ignoring --setuser ceph since I am not root 2024-04-23T18:17:02.430 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: ignoring --setgroup ceph since I am not root 2024-04-23T18:17:02.430 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-mgr[34919]: -- 172.21.15.131:0/153063542 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582e6d8c5a0 con 0x5582e6d61000 2024-04-23T18:17:02.430 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:02.252+0000 7f8aeaf16200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:17:02.430 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:02.435+0000 7f8aeaf16200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-mon[32585]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: ignoring --setuser ceph since I am not root 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: ignoring --setgroup ceph since I am not root 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-mgr[33965]: -- 172.21.15.110:0/47247777 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f00b0785a0 con 0x55f00b04d000 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:02.255+0000 7f632644d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:17:02.434 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:02.439+0000 7f632644d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:02.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1955168103 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0102a00 msgr2=0x7f9ec0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1955168103 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0102a00 0x7f9ec0109320 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009ca0 tx=0x7f9eb802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1955168103 shutdown_connections 2024-04-23T18:17:02.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1955168103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102040 0x7f9ec01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1955168103 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0102a00 0x7f9ec0109320 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1955168103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0100e60 0x7f9ec0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1955168103 >> 172.21.15.26:0/1955168103 conn(0x7f9ec00fc420 msgr2=0x7f9ec00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1955168103 shutdown_connections 2024-04-23T18:17:02.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.529+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1955168103 wait complete. 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 Processor -- start 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 -- start start 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0102040 0x7f9ec019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102a00 0x7f9ec019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.530+0000 7f9ec6dbb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec010c9f0 con 0x7f9ec0100e60 2024-04-23T18:17:02.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec6dbb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec010c870 con 0x7f9ec0102a00 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec6dbb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec010cb70 con 0x7f9ec0102040 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48968/0 (socket says 172.21.15.26:48968) 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 -- 172.21.15.26:0/1817129904 learned_addr learned my addr 172.21.15.26:0/1817129904 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ebffff640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0102040 0x7f9ec019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec5358640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102a00 0x7f9ec019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 -- 172.21.15.26:0/1817129904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0102040 msgr2=0x7f9ec019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0102040 0x7f9ec019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 -- 172.21.15.26:0/1817129904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102a00 msgr2=0x7f9ec019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102a00 0x7f9ec019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 -- 172.21.15.26:0/1817129904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec019fd30 con 0x7f9ec0100e60 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec4b57640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400d950 tx=0x7f9eb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4017020 con 0x7f9ec0100e60 2024-04-23T18:17:02.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec019ffc0 con 0x7f9ec0100e60 2024-04-23T18:17:02.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eb4009c30 con 0x7f9ec0100e60 2024-04-23T18:17:02.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.531+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb400f6b0 con 0x7f9ec0100e60 2024-04-23T18:17:02.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.532+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec01a04e0 con 0x7f9ec0100e60 2024-04-23T18:17:02.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.533+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec0108200 con 0x7f9ec0100e60 2024-04-23T18:17:02.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.533+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f9eb401a020 con 0x7f9ec0100e60 2024-04-23T18:17:02.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.534+0000 7f9ebdffb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 0x7f9eb00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.534+0000 7f9ebffff640 1 -- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 msgr2=0x7f9eb00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:02.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.534+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9eb40cfd60 con 0x7f9ec0100e60 2024-04-23T18:17:02.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.534+0000 7f9ebffff640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 0x7f9eb00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:17:02.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.539+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9eb4021070 con 0x7f9ec0100e60 2024-04-23T18:17:02.693 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:02.541+0000 7f632644d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:17:02.708 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:02.462+0000 7f7bbc404200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:02.708 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:02.574+0000 7f7bbc404200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:17:02.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.729+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9ec002d5c0 con 0x7f9ec0100e60 2024-04-23T18:17:02.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.732+0000 7f9ebdffb640 1 -- 172.21.15.26:0/1817129904 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f9eb4009de0 con 0x7f9ec0100e60 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 msgr2=0x7f9eb00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 0x7f9eb00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 msgr2=0x7f9ec006a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400d950 tx=0x7f9eb400de10 comp rx=0 tx=0).stop 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 shutdown_connections 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f9eb00a3ca0 0x7f9eb00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec0102a00 0x7f9ec019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ec0102040 0x7f9ec019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 --2- 172.21.15.26:0/1817129904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9ec0100e60 0x7f9ec006a190 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 >> 172.21.15.26:0/1817129904 conn(0x7f9ec00fc420 msgr2=0x7f9ec01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 shutdown_connections 2024-04-23T18:17:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:02.734+0000 7f9ec6dbb640 1 -- 172.21.15.26:0/1817129904 wait complete. 2024-04-23T18:17:02.739 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T18:17:02.763 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T18:17:02.763 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T18:17:02.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:02.763 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-04-23T18:17:02.764 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:02.537+0000 7f8aeaf16200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:17:02.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:02.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link delete ceph-brx 2024-04-23T18:17:02.846 INFO:teuthology.orchestra.run.smithi026.stderr:Cannot find device "ceph-brx" 2024-04-23T18:17:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:17:02.847 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-23T18:17:02.847 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T18:17:02.848 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:02.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-23T18:17:03.069 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:02.901+0000 7f7bbc404200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:17:03.069 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:02.992+0000 7f7bbc404200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:17:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:02.850+0000 7f632644d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:02.934+0000 7f632644d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:03 smithi110 ceph-mon[32585]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1173652614' entity='client.admin' 2024-04-23T18:17:03.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:03 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1817129904' entity='client.admin' 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:02.843+0000 7f8aeaf16200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:02.927+0000 7f8aeaf16200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:03 smithi131 ceph-mon[33041]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1173652614' entity='client.admin' 2024-04-23T18:17:03.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:03 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1817129904' entity='client.admin' 2024-04-23T18:17:03.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2035153732' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:17:03.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:03 smithi026 ceph-mon[29364]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-04-23T18:17:03.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1173652614' entity='client.admin' 2024-04-23T18:17:03.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:03 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1817129904' entity='client.admin' 2024-04-23T18:17:04.435 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:04.100+0000 7f8aeaf16200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:17:04.435 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:04.270+0000 7f8aeaf16200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:04.435 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:04.441+0000 7f8aeaf16200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:17:04.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:04.108+0000 7f632644d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:17:04.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:04.277+0000 7f632644d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:04.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:04.446+0000 7f632644d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:17:04.578 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:04.269+0000 7f7bbc404200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:17:04.578 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:04.449+0000 7f7bbc404200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:04.892 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:04.630+0000 7f7bbc404200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:17:04.992 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:04.728+0000 7f8aeaf16200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:17:04.992 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:04.821+0000 7f8aeaf16200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:17:05.019 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:04.731+0000 7f632644d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:17:05.019 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:04.824+0000 7f632644d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:17:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- 172.21.15.26:0/4075778906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 msgr2=0x7faa4c108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 --2- 172.21.15.26:0/4075778906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c108e90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009990 tx=0x7faa3c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- 172.21.15.26:0/4075778906 shutdown_connections 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 --2- 172.21.15.26:0/4075778906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 0x7faa4c10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 --2- 172.21.15.26:0/4075778906 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c108e90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 --2- 172.21.15.26:0/4075778906 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- 172.21.15.26:0/4075778906 >> 172.21.15.26:0/4075778906 conn(0x7faa4c079bf0 msgr2=0x7faa4c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- 172.21.15.26:0/4075778906 shutdown_connections 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- 172.21.15.26:0/4075778906 wait complete. 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 Processor -- start 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.004+0000 7faa52913700 1 -- start start 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 0x7faa4c199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c112cb0 con 0x7faa4c108ab0 2024-04-23T18:17:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c112b30 con 0x7faa4c103480 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa52913700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c112e30 con 0x7faa4c102ac0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa50eb0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa50eb0700 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48988/0 (socket says 172.21.15.26:48988) 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa50eb0700 1 -- 172.21.15.26:0/1415635163 learned_addr learned my addr 172.21.15.26:0/1415635163 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4b7fe700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 0x7faa4c199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 msgr2=0x7faa4c199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 0x7faa4c199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 msgr2=0x7faa4c19d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa4bfff700 1 -- 172.21.15.26:0/1415635163 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c19e110 con 0x7faa4c102ac0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.005+0000 7faa50eb0700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa4bfff700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c199100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0059d0 tx=0x7faa3c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c027070 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c19e3a0 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa3c022470 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1a5c80 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.006+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0175f0 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.007+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c10b040 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.009+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7faa3c02a070 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.009+0000 7faa497fa700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 0x7faa340a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.009+0000 7faa4b7fe700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 msgr2=0x7faa340a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.009+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faa3c0d1550 con 0x7faa4c102ac0 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.009+0000 7faa4b7fe700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 0x7faa340a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:17:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.017+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa3c031020 con 0x7faa4c102ac0 2024-04-23T18:17:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.211+0000 7faa4b7fe700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 msgr2=0x7faa340a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.211+0000 7faa4b7fe700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 0x7faa340a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:17:05.205 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:04.941+0000 7f7bbc404200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:17:05.205 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:05.042+0000 7f7bbc404200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:17:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.273+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faa4c19a460 con 0x7faa4c102ac0 2024-04-23T18:17:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.273+0000 7faa497fa700 1 -- 172.21.15.26:0/1415635163 <== mon.1 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7faa3c013070 con 0x7faa4c102ac0 2024-04-23T18:17:05.267 INFO:teuthology.orchestra.run.smithi026.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-23T18:17:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.275+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 msgr2=0x7faa340a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:17:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.275+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 0x7faa340a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.275+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 msgr2=0x7faa4c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.275+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c199100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0059d0 tx=0x7faa3c0047b0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 shutdown_connections 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7faa340a3c90 0x7faa340a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7faa4c108ab0 0x7faa4c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa4c102ac0 0x7faa4c199100 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 --2- 172.21.15.26:0/1415635163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa4c103480 0x7faa4c199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 >> 172.21.15.26:0/1415635163 conn(0x7faa4c079bf0 msgr2=0x7faa4c100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 shutdown_connections 2024-04-23T18:17:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:05.276+0000 7faa52913700 1 -- 172.21.15.26:0/1415635163 wait complete. 2024-04-23T18:17:05.319 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:05.025+0000 7f632644d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:17:05.319 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:05.216+0000 7f632644d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:05.319 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:05.029+0000 7f8aeaf16200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:17:05.319 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:05.221+0000 7f8aeaf16200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:05.458 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:05.256+0000 7f7bbc404200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:17:05.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:05 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1415635163' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:17:05.692 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:05 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1415635163' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:17:05.741 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:05 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1415635163' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:17:05.958 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:05.466+0000 7f7bbc404200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:17:06.019 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T18:17:06.019 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T18:17:06.019 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi026.front.sepia.ceph.com 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T18:17:06.020 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-23T18:17:06.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:06.020 DEBUG:teuthology.orchestra.run.smithi026:> ip addr 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 ::1/128 scope host 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:17:06.037 INFO:teuthology.orchestra.run.smithi026.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6a brd ff:ff:ff:ff:ff:ff 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f0 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether 0c:c4:7a:6b:ee:6b brd ff:ff:ff:ff:ff:ff 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp7s0f1 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c0 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f0 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: inet 172.21.15.26/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft 42051sec preferred_lft 42051sec 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:17:06.038 INFO:teuthology.orchestra.run.smithi026.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:17:06.039 INFO:teuthology.orchestra.run.smithi026.stdout: link/ether ac:1f:6b:a5:ca:c1 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:17:06.039 INFO:teuthology.orchestra.run.smithi026.stdout: altname enp1s0f1 2024-04-23T18:17:06.039 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T18:17:06.039 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:17:06.039 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-23T18:17:06.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add name ceph-brx type bridge 2024-04-23T18:17:06.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr flush dev ceph-brx 2024-04-23T18:17:06.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set ceph-brx up 2024-04-23T18:17:06.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T18:17:06.040 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-23T18:17:06.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:06.192 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:05.812+0000 7f632644d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:17:06.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:06.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:06.228 DEBUG:teuthology.orchestra.run.smithi026:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T18:17:06.240 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:05.825+0000 7f8aeaf16200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:17:06.241 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:06.108+0000 7f7bbc404200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:17:06.254 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-04-23T18:17:06.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:06.256 DEBUG:teuthology.orchestra.run.smithi026:> route 2024-04-23T18:17:06.323 INFO:teuthology.orchestra.run.smithi026.stdout:Kernel IP routing table 2024-04-23T18:17:06.323 INFO:teuthology.orchestra.run.smithi026.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T18:17:06.323 INFO:teuthology.orchestra.run.smithi026.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-23T18:17:06.323 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-23T18:17:06.323 INFO:teuthology.orchestra.run.smithi026.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T18:17:06.323 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:17:06.323 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-23T18:17:06.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-23T18:17:06.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-23T18:17:06.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-23T18:17:06.324 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-23T18:17:06.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:06.491 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:06.246+0000 7f8aeaf16200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:17:06.492 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:06.342+0000 7f8aeaf16200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:17:06.522 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:06 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:06.229+0000 7f632644d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:17:06.523 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:06 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:06.325+0000 7f632644d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:17:06.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:06.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:06.625 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-04-23T18:17:06.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:06.641 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list-id 2024-04-23T18:17:06.696 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:17:06.696 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-23T18:17:06.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T18:17:06.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-23T18:17:06.696 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-23T18:17:06.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:06.794 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:06.576+0000 7f7bbc404200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:17:06.794 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:06.682+0000 7f7bbc404200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:17:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:06.816 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-23T18:17:06.816 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-23T18:17:06.817 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-23T18:17:06.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:06.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:06.941 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:06 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:06.528+0000 7f632644d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:17:06.946 DEBUG:teuthology.orchestra.run.smithi026:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:17:06.946 DEBUG:teuthology.orchestra.run.smithi026:> set -e 2024-04-23T18:17:06.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set brx.0 up 2024-04-23T18:17:06.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T18:17:06.946 DEBUG:teuthology.orchestra.run.smithi026:> ') 2024-04-23T18:17:06.991 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:06.549+0000 7f8aeaf16200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:17:06.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:07.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:07.026 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:17:07.026 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:07.041 INFO:teuthology.orchestra.run.smithi026.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-23T18:17:07.042 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:17:07.042 DEBUG:teuthology.orchestra.run.smithi026:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:07.098 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-23T18:17:07.098 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-23T18:17:07.098 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/f6f008e6-d2e3-40f3-a7ad-36a6f70dfaf6 2024-04-23T18:17:07.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:07.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/f6f008e6-d2e3-40f3-a7ad-36a6f70dfaf6 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:17:07.160 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:06.904+0000 7f7bbc404200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:17:07.195 INFO:teuthology.orchestra.run.smithi026.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:17:07.195 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: resolved to: "172.21.15.26:3300,172.21.15.110:3300,172.21.15.131:3300" 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: trying mount with new device syntax: 0@2a701120-019c-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/f6f008e6-d2e3-40f3-a7ad-36a6f70dfaf6 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi026.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.26:3300/172.21.15.110:3300/172.21.15.131:3300" will pass to kernel 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi026.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: You just mounted a file system that supports labels which does not 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: applications will generate AVC messages and not be allowed access to 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi026.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-23T18:17:07.309 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T18:17:07.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:17:07.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:07.375 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:17:07.377 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.047+0000 7f632644d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:17:07.377 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.140+0000 7f632644d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:17:07.400 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.070+0000 7f8aeaf16200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:17:07.400 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.163+0000 7f8aeaf16200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:17:07.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:17:07.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:07.475 INFO:teuthology.orchestra.run.smithi026.stdout:client15318 2024-04-23T18:17:07.475 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:17:07.475 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:17:07.492 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:17:07.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:17:07.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:07.591 INFO:teuthology.orchestra.run.smithi026.stdout:client15318 2024-04-23T18:17:07.591 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:17:07.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:17:07.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:07.650 INFO:teuthology.orchestra.run.smithi026.stdout:client15318 2024-04-23T18:17:07.650 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/2a701120-019c-11ef-bc93-c7b262605968.client15318/status') 2024-04-23T18:17:07.691 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.383+0000 7f632644d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:17:07.692 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.516+0000 7f632644d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:17:07.692 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.609+0000 7f632644d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:17:07.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:17:07.709 INFO:teuthology.orchestra.run.smithi026.stdout:instance: client.15318 (3)192.168.0.1:0/2268439987 2024-04-23T18:17:07.710 INFO:teuthology.orchestra.run.smithi026.stdout:blocklisted: false 2024-04-23T18:17:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:0+1 records in 2024-04-23T18:17:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:0+1 records out 2024-04-23T18:17:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:70 bytes copied, 2.5085e-05 s, 2.8 MB/s 2024-04-23T18:17:07.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:17:07.717 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:17:07.727 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:17:07.727 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:17:07.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-23T18:17:07.741 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.406+0000 7f8aeaf16200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:17:07.742 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.539+0000 7f8aeaf16200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:17:07.742 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.635+0000 7f8aeaf16200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:17:07.749 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:07.492+0000 7f7bbc404200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:17:07.749 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:07.593+0000 7f7bbc404200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 -- 172.21.15.26:0/2028131293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8102a00 msgr2=0x7fe3f8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 --2- 172.21.15.26:0/2028131293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8102a00 0x7fe3f8109320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009ca0 tx=0x7fe3f002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 -- 172.21.15.26:0/2028131293 shutdown_connections 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 --2- 172.21.15.26:0/2028131293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102040 0x7fe3f81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 --2- 172.21.15.26:0/2028131293 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8102a00 0x7fe3f8109320 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 --2- 172.21.15.26:0/2028131293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8100e60 0x7fe3f8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 -- 172.21.15.26:0/2028131293 >> 172.21.15.26:0/2028131293 conn(0x7fe3f80fc420 msgr2=0x7fe3f80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 -- 172.21.15.26:0/2028131293 shutdown_connections 2024-04-23T18:17:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.839+0000 7fe3fd831640 1 -- 172.21.15.26:0/2028131293 wait complete. 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 Processor -- start 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 -- start start 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8102040 0x7fe3f819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102a00 0x7fe3f819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810c9f0 con 0x7fe3f8100e60 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810c870 con 0x7fe3f8102a00 2024-04-23T18:17:07.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3fd831640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f810cb70 con 0x7fe3f8102040 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49012/0 (socket says 172.21.15.26:49012) 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f6ffd640 1 -- 172.21.15.26:0/3269801184 learned_addr learned my addr 172.21.15.26:0/3269801184 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f77fe640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102a00 0x7fe3f819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f67fc640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8102040 0x7fe3f819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.840+0000 7fe3f6ffd640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8102040 msgr2=0x7fe3f819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3f6ffd640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8102040 0x7fe3f819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3f6ffd640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102a00 msgr2=0x7fe3f819f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3f6ffd640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102a00 0x7fe3f819f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3f6ffd640 1 -- 172.21.15.26:0/3269801184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f819fd50 con 0x7fe3f8100e60 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3f6ffd640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d950 tx=0x7fe3ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec017020 con 0x7fe3f8100e60 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec009c30 con 0x7fe3f8100e60 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f819ffe0 con 0x7fe3f8100e60 2024-04-23T18:17:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00f6b0 con 0x7fe3f8100e60 2024-04-23T18:17:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.841+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f81a0440 con 0x7fe3f8100e60 2024-04-23T18:17:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.842+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f8108200 con 0x7fe3f8100e60 2024-04-23T18:17:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.843+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fe3ec01a020 con 0x7fe3f8100e60 2024-04-23T18:17:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.844+0000 7fe3fc82f640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 0x7fe3e00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.844+0000 7fe3f67fc640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 msgr2=0x7fe3e00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.844+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe3ec0d0090 con 0x7fe3f8100e60 2024-04-23T18:17:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.844+0000 7fe3f67fc640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 0x7fe3e00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:17:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:07.848+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3ec009de0 con 0x7fe3f8100e60 2024-04-23T18:17:08.000 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:07.862+0000 7f7bbc404200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:17:08.018 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:07.725+0000 7f632644d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.045+0000 7fe3f67fc640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 msgr2=0x7fe3e00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.045+0000 7fe3f67fc640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 0x7fe3e00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:17:08.050 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:07.751+0000 7f8aeaf16200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:08.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.075+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fe3f802d5c0 con 0x7fe3f8100e60 2024-04-23T18:17:08.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.127+0000 7fe3fc82f640 1 -- 172.21.15.26:0/3269801184 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v22) ==== 117+0+0 (secure 0 0 0) 0x7fe3ec09ca20 con 0x7fe3f8100e60 2024-04-23T18:17:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 msgr2=0x7fe3e00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:17:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 0x7fe3e00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 msgr2=0x7fe3f806a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d950 tx=0x7fe3ec00de10 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 shutdown_connections 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7fe3e00a3ca0 0x7fe3e00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3f8102a00 0x7fe3f819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3f8102040 0x7fe3f819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 --2- 172.21.15.26:0/3269801184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe3f8100e60 0x7fe3f806a170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.129+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 >> 172.21.15.26:0/3269801184 conn(0x7fe3f80fc420 msgr2=0x7fe3f81053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.130+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 shutdown_connections 2024-04-23T18:17:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.130+0000 7fe3fd831640 1 -- 172.21.15.26:0/3269801184 wait complete. 2024-04-23T18:17:08.135 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-23T18:17:08.151 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:17:08.160 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:17:08.160 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:17:08.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-23T18:17:08.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 -- 172.21.15.26:0/402036031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14100e60 msgr2=0x7f1f14101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:08.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 --2- 172.21.15.26:0/402036031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14100e60 0x7f1f14101280 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009900 tx=0x7f1f0802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:08.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 -- 172.21.15.26:0/402036031 shutdown_connections 2024-04-23T18:17:08.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 --2- 172.21.15.26:0/402036031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14102a00 0x7f1f14109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 --2- 172.21.15.26:0/402036031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f14102040 0x7f1f141024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 --2- 172.21.15.26:0/402036031 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14100e60 0x7f1f14101280 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 -- 172.21.15.26:0/402036031 >> 172.21.15.26:0/402036031 conn(0x7f1f140fc420 msgr2=0x7f1f140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 -- 172.21.15.26:0/402036031 shutdown_connections 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.287+0000 7f1f19176640 1 -- 172.21.15.26:0/402036031 wait complete. 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 Processor -- start 2024-04-23T18:17:08.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 -- start start 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14100e60 0x7f1f14104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 0x7f1f141034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f14102a00 0x7f1f141039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1410c9f0 con 0x7f1f14102040 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1410c870 con 0x7f1f14100e60 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f19176640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1410cb70 con 0x7f1f14102a00 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f1259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 0x7f1f141034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f1259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 0x7f1f141034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48616/0 (socket says 172.21.15.26:48616) 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f1259c640 1 -- 172.21.15.26:0/3243749901 learned_addr learned my addr 172.21.15.26:0/3243749901 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14100e60 0x7f1f14104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:08.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.288+0000 7f1f1359e640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f14102a00 0x7f1f141039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f14102a00 msgr2=0x7f1f141039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f14102a00 0x7f1f141039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14100e60 msgr2=0x7f1f14104e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14100e60 0x7f1f14104e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f141042b0 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f14100e60 0x7f1f14104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f1259c640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 0x7f1f141034b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400e9f0 tx=0x7f1f0400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0400cd80 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f040054d0 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f141a2310 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04010690 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.289+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f141a27e0 con 0x7f1f14102040 2024-04-23T18:17:08.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.290+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f141a2aa0 con 0x7f1f14102040 2024-04-23T18:17:08.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.291+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f1f04004e90 con 0x7f1f14102040 2024-04-23T18:17:08.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.292+0000 7f1efbfff640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 0x7f1efc0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:08.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.292+0000 7f1f12d9d640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 msgr2=0x7f1efc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:08.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.292+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1f040d0830 con 0x7f1f14102040 2024-04-23T18:17:08.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.292+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 0x7f1efc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:17:08.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.297+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f0409d220 con 0x7f1f14102040 2024-04-23T18:17:08.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:08 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:17:08.442 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:08.023+0000 7f632644d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:17:08.458 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:08.006+0000 7f7bbc404200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:17:08.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:08.110+0000 7f7bbc404200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:17:08.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:08.238+0000 7f7bbc404200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:17:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:08 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:17:08.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.493+0000 7f1f12d9d640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 msgr2=0x7f1efc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:08.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.493+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 0x7f1efc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:17:08.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:08 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:17:08.492 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:08.057+0000 7f8aeaf16200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:17:08.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.511+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1f140fe610 con 0x7f1efc0a3ca0 2024-04-23T18:17:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.894+0000 7f1f12d9d640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 msgr2=0x7f1efc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.26:6800/270207440 2024-04-23T18:17:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:08.894+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 0x7f1efc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:17:08.942 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:08.627+0000 7f632644d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:17:08.942 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:17:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y[33960]: 2024-04-23T18:17:08.844+0000 7f632644d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:17:08.958 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:08.557+0000 7f7bbc404200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:17:08.991 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:08.658+0000 7f8aeaf16200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:17:08.992 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:17:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z[34914]: 2024-04-23T18:17:08.875+0000 7f8aeaf16200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:17:09.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:09.146+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f1f04099190 con 0x7f1f14102040 2024-04-23T18:17:09.427 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.204+0000 7f7bbc404200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: Standby manager daemon y restarted 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: Standby manager daemon y started 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: Standby manager daemon z restarted 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: Standby manager daemon z started 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:17:09.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:17:09.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-mon[29364]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:09.446+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f1f040a36c0 con 0x7f1f14102040 2024-04-23T18:17:09.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:09.447+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 msgr2=0x7f1efc0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:17:09.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:09.447+0000 7f1efbfff640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/270207440,v1:172.21.15.26:6801/270207440] conn(0x7f1efc0a3ca0 0x7f1efc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: Standby manager daemon y restarted 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: Standby manager daemon y started 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: Standby manager daemon z restarted 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: Standby manager daemon z started 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:17:09.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:09 smithi110 ceph-mon[32585]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3269801184' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: Standby manager daemon y restarted 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: Standby manager daemon y started 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.110:0/1017578322' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: Standby manager daemon z restarted 2024-04-23T18:17:09.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: Standby manager daemon z started 2024-04-23T18:17:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:17:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:17:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:17:09.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:09 smithi131 ceph-mon[33041]: from='mgr.? 172.21.15.131:0/839125995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.433+0000 7f7bbc404200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1dd3c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1dd3c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1dd3c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1dd3c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.678 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1dd3c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.679 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1ad36700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.679 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1ad36700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.679 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1ad36700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.679 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1ad36700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:09.679 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:17:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:17:09.683+0000 7f7b1ad36700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:17:10.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:17:10.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: Active manager daemon x restarted 2024-04-23T18:17:10.444 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: Activating manager daemon x 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: mgrmap e21: x(active, starting, since 0.0124371s), standbys: y, z 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:17:10.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:17:10.446 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: Manager daemon x is now available 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.447 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:10 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.456+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mon.0 v2:172.21.15.26:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f1f0404e2f0 con 0x7f1f14102040 2024-04-23T18:17:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.457+0000 7f1efbfff640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1efc035660 0x7f1efc035a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.457+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1f04097440 con 0x7f1efc035660 2024-04-23T18:17:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.459+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1efc035660 0x7f1efc035a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.461+0000 7f1f12d9d640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1efc035660 0x7f1efc035a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009230 tx=0x7f1f080314a0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: Active manager daemon x restarted 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: Activating manager daemon x 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: mgrmap e21: x(active, starting, since 0.0124371s), standbys: y, z 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:17:10.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:17:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:17:10.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: Manager daemon x is now available 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:10 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:17:10.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: Active manager daemon x restarted 2024-04-23T18:17:10.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: Activating manager daemon x 2024-04-23T18:17:10.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: mgrmap e21: x(active, starting, since 0.0124371s), standbys: y, z 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:17:10.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:17:10.494 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: Manager daemon x is now available 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.495 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:10 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:17:10.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.537+0000 7f1efbfff640 1 -- 172.21.15.26:0/3243749901 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1f04097440 con 0x7f1efc035660 2024-04-23T18:17:10.530 INFO:teuthology.orchestra.run.smithi026.stdout:Schedule set for path / 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1efc035660 msgr2=0x7f1efc035a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1efc035660 0x7f1efc035a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009230 tx=0x7f1f080314a0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 msgr2=0x7f1f141034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 --2- 172.21.15.26:0/3243749901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f14102040 0x7f1f141034b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400e9f0 tx=0x7f1f0400edf0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f1359e640 1 -- 172.21.15.26:0/3243749901 reap_dead start 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 shutdown_connections 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 >> 172.21.15.26:0/3243749901 conn(0x7f1f140fc420 msgr2=0x7f1f140fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 shutdown_connections 2024-04-23T18:17:10.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.539+0000 7f1f19176640 1 -- 172.21.15.26:0/3243749901 wait complete. 2024-04-23T18:17:10.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-23T18:17:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1122053133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 msgr2=0x7f5ca41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1122053133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca41024c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0009900 tx=0x7f5ca002a410 comp rx=0 tx=0).stop 2024-04-23T18:17:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1122053133 shutdown_connections 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1122053133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 0x7f5ca4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1122053133 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca41024c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1122053133 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 0x7f5ca4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1122053133 >> 172.21.15.26:0/1122053133 conn(0x7f5ca40fc420 msgr2=0x7f5ca40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1122053133 shutdown_connections 2024-04-23T18:17:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.658+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1122053133 wait complete. 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 Processor -- start 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 -- start start 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 0x7f5ca410bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 0x7f5ca410a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410db00 con 0x7f5ca4102040 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410d980 con 0x7f5ca4102a00 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5cab7bf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410dc80 con 0x7f5ca4100e60 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5ca8d5a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5ca8d5a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48738/0 (socket says 172.21.15.26:48738) 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.659+0000 7f5ca8d5a640 1 -- 172.21.15.26:0/1077901426 learned_addr learned my addr 172.21.15.26:0/1077901426 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca9d5c640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 0x7f5ca410a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca955b640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 0x7f5ca410bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 -- 172.21.15.26:0/1077901426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 msgr2=0x7f5ca410bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 0x7f5ca410bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 -- 172.21.15.26:0/1077901426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 msgr2=0x7f5ca410a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 0x7f5ca410a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 -- 172.21.15.26:0/1077901426 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca410afc0 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5ca8d5a640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0005940 tx=0x7f5ca0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca003f030 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ca0003aa0 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0035070 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca41a2230 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.660+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca41a27d0 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.661+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca4108270 con 0x7f5ca4102040 2024-04-23T18:17:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.662+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f5ca0042030 con 0x7f5ca4102040 2024-04-23T18:17:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.663+0000 7f5c9a7fc640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 0x7f5c900a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.663+0000 7f5ca955b640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 0x7f5c900a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.663+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5ca00f0760 con 0x7f5ca4102040 2024-04-23T18:17:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.664+0000 7f5ca955b640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 0x7f5c900a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4101bb0 tx=0x7f5c94005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:10.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.667+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ca00bd340 con 0x7f5ca4102040 2024-04-23T18:17:10.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.911+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f5ca4062390 con 0x7f5c900a3ae0 2024-04-23T18:17:10.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.980+0000 7f5c9a7fc640 1 -- 172.21.15.26:0/1077901426 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f5ca4062390 con 0x7f5c900a3ae0 2024-04-23T18:17:10.974 INFO:teuthology.orchestra.run.smithi026.stdout:Retention added to path / 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 msgr2=0x7f5c900a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 0x7f5c900a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4101bb0 tx=0x7f5c94005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 msgr2=0x7f5ca410a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0005940 tx=0x7f5ca0031a50 comp rx=0 tx=0).stop 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 shutdown_connections 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5c900a3ae0 0x7f5c900a5fa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca4102a00 0x7f5ca410a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5ca4102040 0x7f5ca410a1f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 --2- 172.21.15.26:0/1077901426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5ca4100e60 0x7f5ca410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 >> 172.21.15.26:0/1077901426 conn(0x7f5ca40fc420 msgr2=0x7f5ca4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:10.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 shutdown_connections 2024-04-23T18:17:10.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:10.982+0000 7f5cab7bf640 1 -- 172.21.15.26:0/1077901426 wait complete. 2024-04-23T18:17:10.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 -- 172.21.15.26:0/2695161954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94101fe0 msgr2=0x7f1b94102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 --2- 172.21.15.26:0/2695161954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94101fe0 0x7f1b94102400 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009900 tx=0x7f1b8402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 -- 172.21.15.26:0/2695161954 shutdown_connections 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 --2- 172.21.15.26:0/2695161954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 --2- 172.21.15.26:0/2695161954 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94103b90 0x7f1b9410a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 --2- 172.21.15.26:0/2695161954 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94101fe0 0x7f1b94102400 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 -- 172.21.15.26:0/2695161954 >> 172.21.15.26:0/2695161954 conn(0x7f1b940fd570 msgr2=0x7f1b940ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 -- 172.21.15.26:0/2695161954 shutdown_connections 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.098+0000 7f1b9990f640 1 -- 172.21.15.26:0/2695161954 wait complete. 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 Processor -- start 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 -- start start 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94101fe0 0x7f1b94198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410db20 con 0x7f1b94103b90 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410d9a0 con 0x7f1b941031d0 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b9990f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410dca0 con 0x7f1b94101fe0 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b937fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b927fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b92ffd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94101fe0 0x7f1b94198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b937fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48748/0 (socket says 172.21.15.26:48748) 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b927fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:46032/0 (socket says 172.21.15.26:46032) 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.099+0000 7f1b937fe640 1 -- 172.21.15.26:0/1171300785 learned_addr learned my addr 172.21.15.26:0/1171300785 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 -- 172.21.15.26:0/1171300785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94101fe0 msgr2=0x7f1b94198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94101fe0 0x7f1b94198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 -- 172.21.15.26:0/1171300785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 msgr2=0x7f1b94198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 -- 172.21.15.26:0/1171300785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9419fd60 con 0x7f1b94103b90 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b927fc640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b937fe640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00eb10 tx=0x7f1b8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004b20 con 0x7f1b94103b90 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b8c004cc0 con 0x7f1b94103b90 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c010670 con 0x7f1b94103b90 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b940741e0 con 0x7f1b94103b90 2024-04-23T18:17:11.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.100+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b94074730 con 0x7f1b94103b90 2024-04-23T18:17:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.101+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94109300 con 0x7f1b94103b90 2024-04-23T18:17:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.102+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f1b8c005020 con 0x7f1b94103b90 2024-04-23T18:17:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.103+0000 7f1b9890d640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 0x7f1b7c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.103+0000 7f1b92ffd640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 0x7f1b7c0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.103+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f1b8c0d0650 con 0x7f1b94103b90 2024-04-23T18:17:11.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.103+0000 7f1b92ffd640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 0x7f1b7c0a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009230 tx=0x7f1b840314a0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.107+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b8c09d1f0 con 0x7f1b94103b90 2024-04-23T18:17:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.360+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f1b94075be0 con 0x7f1b7c0a3a60 2024-04-23T18:17:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.373+0000 7f1b9890d640 1 -- 172.21.15.26:0/1171300785 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f1b94075be0 con 0x7f1b7c0a3a60 2024-04-23T18:17:11.367 INFO:teuthology.orchestra.run.smithi026.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:17:10", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 msgr2=0x7f1b7c0a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 0x7f1b7c0a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009230 tx=0x7f1b840314a0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 msgr2=0x7f1b9419f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00eb10 tx=0x7f1b8c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 shutdown_connections 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1b7c0a3a60 0x7f1b7c0a5f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b941031d0 0x7f1b94198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1b94103b90 0x7f1b9419f6f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 --2- 172.21.15.26:0/1171300785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1b94101fe0 0x7f1b94198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 >> 172.21.15.26:0/1171300785 conn(0x7f1b940fd570 msgr2=0x7f1b941064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 shutdown_connections 2024-04-23T18:17:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.376+0000 7f1b9990f640 1 -- 172.21.15.26:0/1171300785 wait complete. 2024-04-23T18:17:11.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 -- 172.21.15.26:0/2167488388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c101220 msgr2=0x7f314c101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 --2- 172.21.15.26:0/2167488388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c101220 0x7f314c101680 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3140009900 tx=0x7f314002a480 comp rx=0 tx=0).stop 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 -- 172.21.15.26:0/2167488388 shutdown_connections 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 --2- 172.21.15.26:0/2167488388 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101bc0 0x7f314c102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 --2- 172.21.15.26:0/2167488388 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c101220 0x7f314c101680 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 --2- 172.21.15.26:0/2167488388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c1008c0 0x7f314c100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.510+0000 7f315167c640 1 -- 172.21.15.26:0/2167488388 >> 172.21.15.26:0/2167488388 conn(0x7f314c0fc440 msgr2=0x7f314c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.511+0000 7f315167c640 1 -- 172.21.15.26:0/2167488388 shutdown_connections 2024-04-23T18:17:11.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.511+0000 7f315167c640 1 -- 172.21.15.26:0/2167488388 wait complete. 2024-04-23T18:17:11.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.511+0000 7f315167c640 1 Processor -- start 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.511+0000 7f315167c640 1 -- start start 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101220 0x7f314c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 0x7f314c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10fd10 con 0x7f314c1008c0 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10fb90 con 0x7f314c101bc0 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f315167c640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10fe90 con 0x7f314c101220 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314affd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314affd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48772/0 (socket says 172.21.15.26:48772) 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314affd640 1 -- 172.21.15.26:0/504876110 learned_addr learned my addr 172.21.15.26:0/504876110 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314a7fc640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101220 0x7f314c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 0x7f314c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 -- 172.21.15.26:0/504876110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101220 msgr2=0x7f314c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101220 0x7f314c19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 -- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 msgr2=0x7f314c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314b7fe640 1 -- 172.21.15.26:0/504876110 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c1a1d50 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.512+0000 7f314affd640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f314b7fe640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 0x7f314c1a16e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3144009fd0 tx=0x7f314400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314400ce20 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c1a1fe0 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3144005020 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144010690 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.513+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c1a2550 con 0x7f314c101bc0 2024-04-23T18:17:11.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.514+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c10b610 con 0x7f314c101bc0 2024-04-23T18:17:11.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.515+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f31440054d0 con 0x7f314c101bc0 2024-04-23T18:17:11.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.516+0000 7f3133fff640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 0x7f31340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.516+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f31440d0e30 con 0x7f314c101bc0 2024-04-23T18:17:11.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.517+0000 7f314affd640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 0x7f31340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.517+0000 7f314affd640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 0x7f31340a6300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f313c011550 tx=0x7f313c005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.521+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f314409d4a0 con 0x7f314c101bc0 2024-04-23T18:17:11.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:11 smithi026 ceph-mon[29364]: mgrmap e22: x(active, since 1.02055s), standbys: y, z 2024-04-23T18:17:11.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:11 smithi026 ceph-mon[29364]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:11 smithi026 ceph-mon[29364]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:11.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:11 smithi026 ceph-mon[29364]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.746+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f314c062390 con 0x7f31340a3e40 2024-04-23T18:17:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:11 smithi131 ceph-mon[33041]: mgrmap e22: x(active, since 1.02055s), standbys: y, z 2024-04-23T18:17:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:11 smithi131 ceph-mon[33041]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:11 smithi131 ceph-mon[33041]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:11.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:11 smithi131 ceph-mon[33041]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.753+0000 7f3133fff640 1 -- 172.21.15.26:0/504876110 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f314c062390 con 0x7f31340a3e40 2024-04-23T18:17:11.748 INFO:teuthology.orchestra.run.smithi026.stdout:/ 1m 6m3h 2024-04-23T18:17:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 msgr2=0x7f31340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 0x7f31340a6300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f313c011550 tx=0x7f313c005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 msgr2=0x7f314c1a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 0x7f314c1a16e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3144009fd0 tx=0x7f314400ef10 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 shutdown_connections 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f31340a3e40 0x7f31340a6300 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c101bc0 0x7f314c1a16e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f314c101220 0x7f314c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 --2- 172.21.15.26:0/504876110 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f314c1008c0 0x7f314c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 >> 172.21.15.26:0/504876110 conn(0x7f314c0fc440 msgr2=0x7f314c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 shutdown_connections 2024-04-23T18:17:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.756+0000 7f315167c640 1 -- 172.21.15.26:0/504876110 wait complete. 2024-04-23T18:17:11.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-23T18:17:11.788 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-23T18:17:11.799 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-23T18:17:11.817 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:17:11.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-23T18:17:11.830 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-23T18:17:11.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:11.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:11 smithi110 ceph-mon[32585]: mgrmap e22: x(active, since 1.02055s), standbys: y, z 2024-04-23T18:17:11.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:11 smithi110 ceph-mon[32585]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:11 smithi110 ceph-mon[32585]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:11.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:11 smithi110 ceph-mon[32585]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.954+0000 7f5e9ad20640 1 -- 172.21.15.26:0/3500597325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e941029a0 msgr2=0x7f5e941092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.954+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/3500597325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e941029a0 0x7f5e941092c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009ca0 tx=0x7f5e8c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- 172.21.15.26:0/3500597325 shutdown_connections 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/3500597325 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e941029a0 0x7f5e941092c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/3500597325 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e94101fe0 0x7f5e94102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/3500597325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94100e90 0x7f5e941012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- 172.21.15.26:0/3500597325 >> 172.21.15.26:0/3500597325 conn(0x7f5e940fc400 msgr2=0x7f5e940fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- 172.21.15.26:0/3500597325 shutdown_connections 2024-04-23T18:17:11.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- 172.21.15.26:0/3500597325 wait complete. 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 Processor -- start 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- start start 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94101fe0 0x7f5e9419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e941029a0 0x7f5e9419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9410c9d0 con 0x7f5e94100e90 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9410c850 con 0x7f5e94101fe0 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e9ad20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9410cb50 con 0x7f5e941029a0 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e98abc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e98abc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48806/0 (socket says 172.21.15.26:48806) 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.955+0000 7f5e98abc640 1 -- 172.21.15.26:0/2786610454 learned_addr learned my addr 172.21.15.26:0/2786610454 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e93fff640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94101fe0 0x7f5e9419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e992bd640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e941029a0 0x7f5e9419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 -- 172.21.15.26:0/2786610454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e941029a0 msgr2=0x7f5e9419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e941029a0 0x7f5e9419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 -- 172.21.15.26:0/2786610454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94101fe0 msgr2=0x7f5e9419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94101fe0 0x7f5e9419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 -- 172.21.15.26:0/2786610454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9419fd60 con 0x7f5e94100e90 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e98abc640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800ea60 tx=0x7f5e8800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8800cde0 con 0x7f5e94100e90 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e880035f0 con 0x7f5e94100e90 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9419fff0 con 0x7f5e94100e90 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88010690 con 0x7f5e94100e90 2024-04-23T18:17:11.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.956+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e941a04b0 con 0x7f5e94100e90 2024-04-23T18:17:11.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.957+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e941081a0 con 0x7f5e94100e90 2024-04-23T18:17:11.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.958+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f5e880037a0 con 0x7f5e94100e90 2024-04-23T18:17:11.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.959+0000 7f5e91ffb640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 0x7f5e840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.959+0000 7f5e93fff640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 0x7f5e840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.959+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5e880d0a60 con 0x7f5e94100e90 2024-04-23T18:17:11.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.960+0000 7f5e93fff640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 0x7f5e840a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5e800095f0 tx=0x7f5e80005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:11.964+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e8809d130 con 0x7f5e94100e90 2024-04-23T18:17:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.207+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e94062bf0 con 0x7f5e94100e90 2024-04-23T18:17:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.209+0000 7f5e91ffb640 1 -- 172.21.15.26:0/2786610454 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23037 (secure 0 0 0) 0x7f5e8809ca30 con 0x7f5e94100e90 2024-04-23T18:17:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:12.204 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:17:09.435910+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 msgr2=0x7f5e840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 0x7f5e840a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5e800095f0 tx=0x7f5e80005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 msgr2=0x7f5e94069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800ea60 tx=0x7f5e8800ef20 comp rx=0 tx=0).stop 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 shutdown_connections 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e840a3d20 0x7f5e840a61e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e94101fe0 0x7f5e9419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e941029a0 0x7f5e9419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 --2- 172.21.15.26:0/2786610454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e94100e90 0x7f5e94069cb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 >> 172.21.15.26:0/2786610454 conn(0x7f5e940fc400 msgr2=0x7f5e94105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 shutdown_connections 2024-04-23T18:17:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.212+0000 7f5e9ad20640 1 -- 172.21.15.26:0/2786610454 wait complete. 2024-04-23T18:17:12.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:17:12.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:17:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 -- 172.21.15.26:0/2195003674 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec103b60 msgr2=0x7f6bec10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/2195003674 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec103b60 0x7f6bec10a3f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009ca0 tx=0x7f6be402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 -- 172.21.15.26:0/2195003674 shutdown_connections 2024-04-23T18:17:12.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/2195003674 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec103b60 0x7f6bec10a3f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/2195003674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec1031a0 0x7f6bec103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/2195003674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec101fb0 0x7f6bec1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 -- 172.21.15.26:0/2195003674 >> 172.21.15.26:0/2195003674 conn(0x7f6bec0fd520 msgr2=0x7f6bec0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 -- 172.21.15.26:0/2195003674 shutdown_connections 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.335+0000 7f6bf21cd640 1 -- 172.21.15.26:0/2195003674 wait complete. 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 Processor -- start 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 -- start start 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec1031a0 0x7f6bec074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec103b60 0x7f6bec074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10daf0 con 0x7f6bec101fb0 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10d970 con 0x7f6bec1031a0 2024-04-23T18:17:12.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bf21cd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10dc70 con 0x7f6bec103b60 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6beb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6beb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48826/0 (socket says 172.21.15.26:48826) 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6beb7fe640 1 -- 172.21.15.26:0/4247662208 learned_addr learned my addr 172.21.15.26:0/4247662208 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6beaffd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec1031a0 0x7f6bec074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.336+0000 7f6bebfff640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec103b60 0x7f6bec074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 -- 172.21.15.26:0/4247662208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec103b60 msgr2=0x7f6bec074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec103b60 0x7f6bec074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 -- 172.21.15.26:0/4247662208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec1031a0 msgr2=0x7f6bec074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec1031a0 0x7f6bec074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 -- 172.21.15.26:0/4247662208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec075110 con 0x7f6bec101fb0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6beb7fe640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00d770 tx=0x7f6bdc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc017070 con 0x7f6bec101fb0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1a2280 con 0x7f6bec101fb0 2024-04-23T18:17:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bdc009c30 con 0x7f6bec101fb0 2024-04-23T18:17:12.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc003cc0 con 0x7f6bec101fb0 2024-04-23T18:17:12.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.337+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec1a27d0 con 0x7f6bec101fb0 2024-04-23T18:17:12.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.338+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec109350 con 0x7f6bec101fb0 2024-04-23T18:17:12.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.339+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f6bdc003e60 con 0x7f6bec101fb0 2024-04-23T18:17:12.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.340+0000 7f6be8ff9640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 0x7f6bd40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.340+0000 7f6beaffd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 0x7f6bd40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.340+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6bdc0d0550 con 0x7f6bec101fb0 2024-04-23T18:17:12.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.340+0000 7f6beaffd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 0x7f6bd40a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6bec075a70 tx=0x7f6be000b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.344+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bdc009de0 con 0x7f6bec101fb0 2024-04-23T18:17:12.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.537+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6bec062b50 con 0x7f6bec101fb0 2024-04-23T18:17:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.537+0000 7f6be8ff9640 1 -- 172.21.15.26:0/4247662208 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f6bdc09c570 con 0x7f6bec101fb0 2024-04-23T18:17:12.532 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 msgr2=0x7f6bd40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 0x7f6bd40a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6bec075a70 tx=0x7f6be000b390 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 msgr2=0x7f6bec075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00d770 tx=0x7f6bdc00dc30 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 shutdown_connections 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6bd40a3d20 0x7f6bd40a61e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec1031a0 0x7f6bec074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bec103b60 0x7f6bec074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 --2- 172.21.15.26:0/4247662208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6bec101fb0 0x7f6bec075d00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 >> 172.21.15.26:0/4247662208 conn(0x7f6bec0fd520 msgr2=0x7f6bec106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 shutdown_connections 2024-04-23T18:17:12.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.540+0000 7f6bf21cd640 1 -- 172.21.15.26:0/4247662208 wait complete. 2024-04-23T18:17:12.545 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:17:11] ENGINE Bus STARTING 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:17:11] ENGINE Serving on http://172.21.15.26:8765 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:17:11] ENGINE Serving on https://172.21.15.26:7150 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: [23/Apr/2024:18:17:11] ENGINE Bus STARTED 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.547 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:12 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2786610454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:12.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:17:12.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 -- 172.21.15.26:0/2195190334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4103190 msgr2=0x7ffbb4103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 --2- 172.21.15.26:0/2195190334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4103190 0x7ffbb4103610 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009900 tx=0x7ffba802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 -- 172.21.15.26:0/2195190334 shutdown_connections 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 --2- 172.21.15.26:0/2195190334 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 0x7ffbb410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 --2- 172.21.15.26:0/2195190334 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4103190 0x7ffbb4103610 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 --2- 172.21.15.26:0/2195190334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4101fa0 0x7ffbb41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 -- 172.21.15.26:0/2195190334 >> 172.21.15.26:0/2195190334 conn(0x7ffbb40fd530 msgr2=0x7ffbb40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 -- 172.21.15.26:0/2195190334 shutdown_connections 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.661+0000 7ffbba660640 1 -- 172.21.15.26:0/2195190334 wait complete. 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 Processor -- start 2024-04-23T18:17:12.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 -- start start 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 0x7ffbb4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 0x7ffbb4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb410dae0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb410d960 con 0x7ffbb4103190 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbba660640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb410dc60 con 0x7ffbb4103b50 2024-04-23T18:17:12.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.662+0000 7ffbb3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48838/0 (socket says 172.21.15.26:48838) 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 -- 172.21.15.26:0/2721748206 learned_addr learned my addr 172.21.15.26:0/2721748206 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb37fe640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 0x7ffbb4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb8bfd640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 0x7ffbb4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 -- 172.21.15.26:0/2721748206 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 msgr2=0x7ffbb4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 0x7ffbb4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 -- 172.21.15.26:0/2721748206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 msgr2=0x7ffbb4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 0x7ffbb4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 -- 172.21.15.26:0/2721748206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb4075110 con 0x7ffbb4101fa0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb37fe640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 0x7ffbb4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb3fff640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffba400e9c0 tx=0x7ffba400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba400cd80 con 0x7ffbb4101fa0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba40035f0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4010690 con 0x7ffbb4101fa0 2024-04-23T18:17:12.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb41a2290 con 0x7ffbb4101fa0 2024-04-23T18:17:12.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.663+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb41a2800 con 0x7ffbb4101fa0 2024-04-23T18:17:12.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.665+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb4068d70 con 0x7ffbb4101fa0 2024-04-23T18:17:12.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.665+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7ffba40037a0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.666+0000 7ffbb17fa640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 0x7ffb9c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.666+0000 7ffbb37fe640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 0x7ffb9c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.666+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ffba40d0b10 con 0x7ffbb4101fa0 2024-04-23T18:17:12.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.667+0000 7ffbb37fe640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 0x7ffb9c0a6260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffba8000c00 tx=0x7ffba80314a0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.671+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffba409d1e0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: [23/Apr/2024:18:17:11] ENGINE Bus STARTING 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: [23/Apr/2024:18:17:11] ENGINE Serving on http://172.21.15.26:8765 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: [23/Apr/2024:18:17:11] ENGINE Serving on https://172.21.15.26:7150 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: [23/Apr/2024:18:17:11] ENGINE Bus STARTED 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.742 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.743 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:12 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2786610454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.891+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffbb41092c0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.891+0000 7ffbb17fa640 1 -- 172.21.15.26:0/2721748206 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ffbb41092c0 con 0x7ffbb4101fa0 2024-04-23T18:17:12.886 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 msgr2=0x7ffb9c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 0x7ffb9c0a6260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffba8000c00 tx=0x7ffba80314a0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 msgr2=0x7ffbb4075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffba400e9c0 tx=0x7ffba400ee80 comp rx=0 tx=0).stop 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 shutdown_connections 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffb9c0a3da0 0x7ffb9c0a6260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbb4103190 0x7ffbb4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffbb4103b50 0x7ffbb4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 --2- 172.21.15.26:0/2721748206 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffbb4101fa0 0x7ffbb4075d00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.893+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 >> 172.21.15.26:0/2721748206 conn(0x7ffbb40fd530 msgr2=0x7ffbb4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.894+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 shutdown_connections 2024-04-23T18:17:12.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:12.894+0000 7ffbba660640 1 -- 172.21.15.26:0/2721748206 wait complete. 2024-04-23T18:17:12.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: [23/Apr/2024:18:17:11] ENGINE Bus STARTING 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: [23/Apr/2024:18:17:11] ENGINE Serving on http://172.21.15.26:8765 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: [23/Apr/2024:18:17:11] ENGINE Serving on https://172.21.15.26:7150 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: [23/Apr/2024:18:17:11] ENGINE Bus STARTED 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:12.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:17:12.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:12 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2786610454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 -- 172.21.15.26:0/2064584886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 msgr2=0x7fd7281023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/2064584886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd7281023b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd724009900 tx=0x7fd72402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 -- 172.21.15.26:0/2064584886 shutdown_connections 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/2064584886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/2064584886 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 0x7fd728103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/2064584886 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd7281023b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 -- 172.21.15.26:0/2064584886 >> 172.21.15.26:0/2064584886 conn(0x7fd7280fd500 msgr2=0x7fd7280ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 -- 172.21.15.26:0/2064584886 shutdown_connections 2024-04-23T18:17:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.013+0000 7fd72fb5d640 1 -- 172.21.15.26:0/2064584886 wait complete. 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 Processor -- start 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 -- start start 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 0x7fd728198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810da90 con 0x7fd728101f90 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810d910 con 0x7fd728103b40 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72fb5d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810dc10 con 0x7fd728103180 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48850/0 (socket says 172.21.15.26:48850) 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 -- 172.21.15.26:0/1017510624 learned_addr learned my addr 172.21.15.26:0/1017510624 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72e0fa640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d0f8640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 0x7fd728198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 -- 172.21.15.26:0/1017510624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 msgr2=0x7fd728198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 0x7fd728198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 -- 172.21.15.26:0/1017510624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 msgr2=0x7fd72819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.014+0000 7fd72d8f9640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd72d8f9640 1 -- 172.21.15.26:0/1017510624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72819fe00 con 0x7fd728101f90 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd72e0fa640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd72d8f9640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd724006010 tx=0x7fd7240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72403f030 con 0x7fd728101f90 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7240318f0 con 0x7fd728101f90 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7281a0090 con 0x7fd728101f90 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724035070 con 0x7fd728101f90 2024-04-23T18:17:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.015+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7281a05d0 con 0x7fd728101f90 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.016+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd728109420 con 0x7fd728101f90 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.017+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fd724042030 con 0x7fd728101f90 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.018+0000 7fd71effd640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 0x7fd7140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.018+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd7240f0c40 con 0x7fd728101f90 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.018+0000 7fd72d0f8640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 0x7fd7140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.019+0000 7fd72d0f8640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 0x7fd7140a6260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7281996f0 tx=0x7fd71800b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.025+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7240bccc0 con 0x7fd728101f90 2024-04-23T18:17:13.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.220+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd728062b50 con 0x7fd728101f90 2024-04-23T18:17:13.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.220+0000 7fd71effd640 1 -- 172.21.15.26:0/1017510624 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fd7240c0fc0 con 0x7fd728101f90 2024-04-23T18:17:13.215 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:17:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 msgr2=0x7fd7140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 0x7fd7140a6260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7281996f0 tx=0x7fd71800b390 comp rx=0 tx=0).stop 2024-04-23T18:17:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 msgr2=0x7fd728198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd724006010 tx=0x7fd7240042f0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 shutdown_connections 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd7140a3da0 0x7fd7140a6260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd728103b40 0x7fd72819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd728103180 0x7fd728198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.222+0000 7fd72fb5d640 1 --2- 172.21.15.26:0/1017510624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd728101f90 0x7fd728198230 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.223+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 >> 172.21.15.26:0/1017510624 conn(0x7fd7280fd500 msgr2=0x7fd7281065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.223+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 shutdown_connections 2024-04-23T18:17:13.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.223+0000 7fd72fb5d640 1 -- 172.21.15.26:0/1017510624 wait complete. 2024-04-23T18:17:13.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:13.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 -- 172.21.15.26:0/1817365896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 msgr2=0x7fac7410a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/1817365896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac7410a120 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009ca0 tx=0x7fac6c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 -- 172.21.15.26:0/1817365896 shutdown_connections 2024-04-23T18:17:13.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/1817365896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 0x7fac741077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/1817365896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac7410a120 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/1817365896 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 0x7fac74104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 -- 172.21.15.26:0/1817365896 >> 172.21.15.26:0/1817365896 conn(0x7fac740fc400 msgr2=0x7fac740fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 -- 172.21.15.26:0/1817365896 shutdown_connections 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.338+0000 7fac7a5fe640 1 -- 172.21.15.26:0/1817365896 wait complete. 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 Processor -- start 2024-04-23T18:17:13.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 -- start start 2024-04-23T18:17:13.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 0x7fac74102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 0x7fac74100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fac7410dae0 con 0x7fac74107d30 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac7410d960 con 0x7fac74105400 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.339+0000 7fac7a5fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fac7410dc60 con 0x7fac74102ad0 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac737fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 0x7fac74100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48874/0 (socket says 172.21.15.26:48874) 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac73fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 0x7fac74102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 -- 172.21.15.26:0/127271523 learned_addr learned my addr 172.21.15.26:0/127271523 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 -- 172.21.15.26:0/127271523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 msgr2=0x7fac74102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 0x7fac74102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 -- 172.21.15.26:0/127271523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 msgr2=0x7fac74100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 0x7fac74100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 -- 172.21.15.26:0/127271523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac74101850 con 0x7fac74107d30 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac78b9b640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fac6c0097b0 tx=0x7fac6c031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c003b70 con 0x7fac74107d30 2024-04-23T18:17:13.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac6c003d10 con 0x7fac74107d30 2024-04-23T18:17:13.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac741a22e0 con 0x7fac74107d30 2024-04-23T18:17:13.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c02fc10 con 0x7fac74107d30 2024-04-23T18:17:13.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.340+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac741a2830 con 0x7fac74107d30 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.341+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac740fdef0 con 0x7fac74107d30 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.343+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fac6c004040 con 0x7fac74107d30 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.343+0000 7fac717fa640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 0x7fac5c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.343+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fac6c0f1de0 con 0x7fac74107d30 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.343+0000 7fac73fff640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 0x7fac5c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.344+0000 7fac73fff640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 0x7fac5c0a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac64009bd0 tx=0x7fac64005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.347+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac6c0be5c0 con 0x7fac74107d30 2024-04-23T18:17:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.563+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac74062b50 con 0x7fac74107d30 2024-04-23T18:17:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.565+0000 7fac717fa640 1 -- 172.21.15.26:0/127271523 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23037 (secure 0 0 0) 0x7fac6c0bdec0 con 0x7fac74107d30 2024-04-23T18:17:13.560 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:13.560 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:17:09.435910+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.567+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 msgr2=0x7fac5c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.567+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 0x7fac5c0a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac64009bd0 tx=0x7fac64005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.567+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 msgr2=0x7fac74100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.567+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fac6c0097b0 tx=0x7fac6c031770 comp rx=0 tx=0).stop 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 shutdown_connections 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fac5c0a3d20 0x7fac5c0a61e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac74105400 0x7fac74100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fac74107d30 0x7fac74100fc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 --2- 172.21.15.26:0/127271523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac74102ad0 0x7fac74102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 >> 172.21.15.26:0/127271523 conn(0x7fac740fc400 msgr2=0x7fac74103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 shutdown_connections 2024-04-23T18:17:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.568+0000 7fac7a5fe640 1 -- 172.21.15.26:0/127271523 wait complete. 2024-04-23T18:17:13.574 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:17:13.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:17:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 -- 172.21.15.26:0/3434664191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 msgr2=0x7f97a410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 --2- 172.21.15.26:0/3434664191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a410a3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f979c009ca0 tx=0x7f979c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 -- 172.21.15.26:0/3434664191 shutdown_connections 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 --2- 172.21.15.26:0/3434664191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a410a3e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 --2- 172.21.15.26:0/3434664191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 0x7f97a4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 --2- 172.21.15.26:0/3434664191 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 -- 172.21.15.26:0/3434664191 >> 172.21.15.26:0/3434664191 conn(0x7f97a40fd530 msgr2=0x7f97a40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 -- 172.21.15.26:0/3434664191 shutdown_connections 2024-04-23T18:17:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.690+0000 7f97aaebb640 1 -- 172.21.15.26:0/3434664191 wait complete. 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 Processor -- start 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 -- start start 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 0x7f97a419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a41a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410daa0 con 0x7f97a4101fa0 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410d920 con 0x7f97a4103b50 2024-04-23T18:17:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97aaebb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f97a410dc20 con 0x7f97a4103190 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48890/0 (socket says 172.21.15.26:48890) 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 -- 172.21.15.26:0/870766346 learned_addr learned my addr 172.21.15.26:0/870766346 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a3fff640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 0x7f97a419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a9458640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a41a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 -- 172.21.15.26:0/870766346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 msgr2=0x7f97a419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 0x7f97a419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 -- 172.21.15.26:0/870766346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 msgr2=0x7f97a41a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a41a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.691+0000 7f97a8c57640 1 -- 172.21.15.26:0/870766346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41a6390 con 0x7f97a4101fa0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97a8c57640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f979800e9c0 tx=0x7f979800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979800cd80 con 0x7f97a4101fa0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41a6680 con 0x7f97a4101fa0 2024-04-23T18:17:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97980035f0 con 0x7f97a4101fa0 2024-04-23T18:17:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798010690 con 0x7f97a4101fa0 2024-04-23T18:17:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.692+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41a6c20 con 0x7f97a4101fa0 2024-04-23T18:17:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.693+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a41a6ee0 con 0x7f97a4101fa0 2024-04-23T18:17:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.694+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f97980037a0 con 0x7f97a4101fa0 2024-04-23T18:17:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.695+0000 7f97a1ffb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 0x7f97940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.695+0000 7f97a3fff640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 0x7f97940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.695+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f97980d0b90 con 0x7f97a4101fa0 2024-04-23T18:17:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.696+0000 7f97a3fff640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 0x7f97940a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97a419fc80 tx=0x7f9790005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.700+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f979809d200 con 0x7f97a4101fa0 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4247662208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.885 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2721748206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.886 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:17:13.887 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1017510624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.890+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97a41a70b0 con 0x7f97a4101fa0 2024-04-23T18:17:13.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.891+0000 7f97a1ffb640 1 -- 172.21.15.26:0/870766346 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f979809cb00 con 0x7f97a4101fa0 2024-04-23T18:17:13.888 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:17:13.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 msgr2=0x7f97940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 0x7f97940a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97a419fc80 tx=0x7f9790005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 msgr2=0x7f97a419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f979800e9c0 tx=0x7f979800ee80 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 shutdown_connections 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f97940a3d20 0x7f97940a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97a4103b50 0x7f97a41a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97a4103190 0x7f97a419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 --2- 172.21.15.26:0/870766346 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f97a4101fa0 0x7f97a419e770 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 >> 172.21.15.26:0/870766346 conn(0x7f97a40fd530 msgr2=0x7f97a4074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 shutdown_connections 2024-04-23T18:17:13.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:13.893+0000 7f97aaebb640 1 -- 172.21.15.26:0/870766346 wait complete. 2024-04-23T18:17:13.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4247662208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.901 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2721748206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:17:13.903 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1017510624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4247662208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2721748206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:17:13.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:13.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:13.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Updating smithi026:/etc/ceph/ceph.conf 2024-04-23T18:17:13.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-23T18:17:13.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: Updating smithi131:/etc/ceph/ceph.conf 2024-04-23T18:17:13.945 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1017510624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 -- 172.21.15.26:0/703501817 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40101080 msgr2=0x7f5a40101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 --2- 172.21.15.26:0/703501817 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40101080 0x7f5a40101460 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009900 tx=0x7f5a3402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 -- 172.21.15.26:0/703501817 shutdown_connections 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 --2- 172.21.15.26:0/703501817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101a30 0x7f5a40105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 --2- 172.21.15.26:0/703501817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40106150 0x7f5a4010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 --2- 172.21.15.26:0/703501817 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40101080 0x7f5a40101460 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 -- 172.21.15.26:0/703501817 >> 172.21.15.26:0/703501817 conn(0x7f5a40078000 msgr2=0x7f5a40078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.025+0000 7f5a47031640 1 -- 172.21.15.26:0/703501817 shutdown_connections 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 -- 172.21.15.26:0/703501817 wait complete. 2024-04-23T18:17:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 Processor -- start 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 -- start start 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 0x7f5a401994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 0x7f5a401a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40111090 con 0x7f5a40106150 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40110f10 con 0x7f5a40101080 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a47031640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5a40111210 con 0x7f5a40199fc0 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.026+0000 7f5a3ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48908/0 (socket says 172.21.15.26:48908) 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a455ce640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 0x7f5a401a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 -- 172.21.15.26:0/493618554 learned_addr learned my addr 172.21.15.26:0/493618554 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a44dcd640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 0x7f5a401994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 -- 172.21.15.26:0/493618554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 msgr2=0x7f5a401a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 0x7f5a401a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 -- 172.21.15.26:0/493618554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 msgr2=0x7f5a401994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 0x7f5a401994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 -- 172.21.15.26:0/493618554 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a401a4d40 con 0x7f5a40106150 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a44dcd640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 0x7f5a401994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a455ce640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 0x7f5a401a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3ffff640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9c0 tx=0x7f5a2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00cd80 con 0x7f5a40106150 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2c0054d0 con 0x7f5a40106150 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a401a4fd0 con 0x7f5a40106150 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c010430 con 0x7f5a40106150 2024-04-23T18:17:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.027+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a401a54f0 con 0x7f5a40106150 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.029+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a40068d70 con 0x7f5a40106150 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.030+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5a2c004fd0 con 0x7f5a40106150 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.030+0000 7f5a3dffb640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 0x7f5a300a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.030+0000 7f5a44dcd640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 0x7f5a300a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.030+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5a2c0d0d50 con 0x7f5a40106150 2024-04-23T18:17:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.031+0000 7f5a44dcd640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 0x7f5a300a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a34031440 tx=0x7f5a34005b50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.035+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a2c0d4030 con 0x7f5a40106150 2024-04-23T18:17:14.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.230+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a40101a30 con 0x7f5a40106150 2024-04-23T18:17:14.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.230+0000 7f5a3dffb640 1 -- 172.21.15.26:0/493618554 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f5a2c09d2b0 con 0x7f5a40106150 2024-04-23T18:17:14.225 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 msgr2=0x7f5a300a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 0x7f5a300a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a34031440 tx=0x7f5a34005b50 comp rx=0 tx=0).stop 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 msgr2=0x7f5a40199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9c0 tx=0x7f5a2c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 shutdown_connections 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5a300a3db0 0x7f5a300a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a40199fc0 0x7f5a401a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5a40106150 0x7f5a40199a30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 --2- 172.21.15.26:0/493618554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a40101080 0x7f5a401994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 >> 172.21.15.26:0/493618554 conn(0x7f5a40078000 msgr2=0x7f5a400ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 shutdown_connections 2024-04-23T18:17:14.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.232+0000 7f5a47031640 1 -- 172.21.15.26:0/493618554 wait complete. 2024-04-23T18:17:14.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.349+0000 7f40b939b640 1 -- 172.21.15.26:0/137896647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b41031d0 msgr2=0x7f40b41035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.349+0000 7f40b939b640 1 --2- 172.21.15.26:0/137896647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b41031d0 0x7f40b41035b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009900 tx=0x7f40a402a510 comp rx=0 tx=0).stop 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- 172.21.15.26:0/137896647 shutdown_connections 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- 172.21.15.26:0/137896647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b4103af0 0x7f40b410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- 172.21.15.26:0/137896647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b410c510 0x7f40b410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- 172.21.15.26:0/137896647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b41031d0 0x7f40b41035b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- 172.21.15.26:0/137896647 >> 172.21.15.26:0/137896647 conn(0x7f40b4077f50 msgr2=0x7f40b4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- 172.21.15.26:0/137896647 shutdown_connections 2024-04-23T18:17:14.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- 172.21.15.26:0/137896647 wait complete. 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 Processor -- start 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- start start 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 0x7f40b419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b4103af0 0x7f40b419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.350+0000 7f40b939b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f40b41121a0 con 0x7f40b410c510 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b939b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4112020 con 0x7f40b41031d0 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b939b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4112320 con 0x7f40b4103af0 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48924/0 (socket says 172.21.15.26:48924) 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 -- 172.21.15.26:0/3881738296 learned_addr learned my addr 172.21.15.26:0/3881738296 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b27fc640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b4103af0 0x7f40b419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b2ffd640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 0x7f40b419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 -- 172.21.15.26:0/3881738296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b4103af0 msgr2=0x7f40b419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b4103af0 0x7f40b419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 -- 172.21.15.26:0/3881738296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 msgr2=0x7f40b419e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 0x7f40b419e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 -- 172.21.15.26:0/3881738296 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4198d90 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b2ffd640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 0x7f40b419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b37fe640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00eaa0 tx=0x7f40ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac00cdb0 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40ac005020 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b4199080 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0116d0 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.351+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b41a5320 con 0x7f40b410c510 2024-04-23T18:17:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.352+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b4068d70 con 0x7f40b410c510 2024-04-23T18:17:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.353+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f40ac0051d0 con 0x7f40b410c510 2024-04-23T18:17:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.354+0000 7f409bfff640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 0x7f409c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.354+0000 7f40b2ffd640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 0x7f409c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.354+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f40ac0d0b30 con 0x7f40b410c510 2024-04-23T18:17:14.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.355+0000 7f40b2ffd640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 0x7f409c0a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a4031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.359+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40ac016030 con 0x7f40b410c510 2024-04-23T18:17:14.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.551+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40b41076c0 con 0x7f40b410c510 2024-04-23T18:17:14.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.552+0000 7f409bfff640 1 -- 172.21.15.26:0/3881738296 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f40ac09d030 con 0x7f40b410c510 2024-04-23T18:17:14.546 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:17:14.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 msgr2=0x7f409c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 0x7f409c0a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a4031400 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 msgr2=0x7f40b4198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00eaa0 tx=0x7f40ac00ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 shutdown_connections 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f409c0a3db0 0x7f409c0a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f40b410c510 0x7f40b4198610 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40b4103af0 0x7f40b419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 --2- 172.21.15.26:0/3881738296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40b41031d0 0x7f40b419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 >> 172.21.15.26:0/3881738296 conn(0x7f40b4077f50 msgr2=0x7f40b40ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 shutdown_connections 2024-04-23T18:17:14.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.554+0000 7f40b939b640 1 -- 172.21.15.26:0/3881738296 wait complete. 2024-04-23T18:17:14.560 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-23T18:17:14.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:17:14.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 -- 172.21.15.26:0/1179915750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 msgr2=0x7fbce010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 --2- 172.21.15.26:0/1179915750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce010dd30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009ca0 tx=0x7fbcd802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 -- 172.21.15.26:0/1179915750 shutdown_connections 2024-04-23T18:17:14.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 --2- 172.21.15.26:0/1179915750 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce010dd30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 --2- 172.21.15.26:0/1179915750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101a40 0x7fbce0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 --2- 172.21.15.26:0/1179915750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce0101090 0x7fbce0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 -- 172.21.15.26:0/1179915750 >> 172.21.15.26:0/1179915750 conn(0x7fbce00771d0 msgr2=0x7fbce00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 -- 172.21.15.26:0/1179915750 shutdown_connections 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.667+0000 7fbce63af640 1 -- 172.21.15.26:0/1179915750 wait complete. 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 Processor -- start 2024-04-23T18:17:14.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 -- start start 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 0x7fbce01a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce019a760 0x7fbce019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01110b0 con 0x7fbce0106160 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0110f30 con 0x7fbce019a760 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce63af640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0111230 con 0x7fbce0101090 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48932/0 (socket says 172.21.15.26:48932) 2024-04-23T18:17:14.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbcdf7fe640 1 -- 172.21.15.26:0/4145789417 learned_addr learned my addr 172.21.15.26:0/4145789417 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.668+0000 7fbce494c640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce019a760 0x7fbce019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdffff640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 0x7fbce01a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 -- 172.21.15.26:0/4145789417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 msgr2=0x7fbce01a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 0x7fbce01a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 -- 172.21.15.26:0/4145789417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce019a760 msgr2=0x7fbce019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce019a760 0x7fbce019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 -- 172.21.15.26:0/4145789417 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce019b360 con 0x7fbce0106160 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdffff640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 0x7fbce01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdf7fe640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8005fd0 tx=0x7fbcc800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8017070 con 0x7fbce0106160 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcc8009c30 con 0x7fbce0106160 2024-04-23T18:17:14.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce01a7440 con 0x7fbce0106160 2024-04-23T18:17:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8003d30 con 0x7fbce0106160 2024-04-23T18:17:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.669+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01a7840 con 0x7fbce0106160 2024-04-23T18:17:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.670+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce0069e00 con 0x7fbce0106160 2024-04-23T18:17:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.671+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fbcc8003ed0 con 0x7fbce0106160 2024-04-23T18:17:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.672+0000 7fbcdd7fa640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 0x7fbccc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.672+0000 7fbcdffff640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 0x7fbccc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.672+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fbcc80d0590 con 0x7fbce0106160 2024-04-23T18:17:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.673+0000 7fbcdffff640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 0x7fbccc0a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009b50 tx=0x7fbcd0005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.677+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcc8020030 con 0x7fbce0106160 2024-04-23T18:17:14.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.859+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbce0101a40 con 0x7fbccc0a3db0 2024-04-23T18:17:14.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.867+0000 7fbcdd7fa640 1 -- 172.21.15.26:0/4145789417 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257734 (secure 0 0 0) 0x7fbce0101a40 con 0x7fbccc0a3db0 2024-04-23T18:17:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:14.865 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 msgr2=0x7fbccc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 0x7fbccc0a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009b50 tx=0x7fbcd0005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 msgr2=0x7fbce01a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8005fd0 tx=0x7fbcc800dde0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 shutdown_connections 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3db0 0x7fbccc0a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce019a760 0x7fbce019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce0106160 0x7fbce01a0be0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 --2- 172.21.15.26:0/4145789417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce0101090 0x7fbce01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 >> 172.21.15.26:0/4145789417 conn(0x7fbce00771d0 msgr2=0x7fbce00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 shutdown_connections 2024-04-23T18:17:14.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:14.870+0000 7fbce63af640 1 -- 172.21.15.26:0/4145789417 wait complete. 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/127271523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/870766346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:14.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/493618554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.877 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:17:13.453977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189190,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":972,"num_read_kb":648,"num_write":469,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181392,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401300,"kb_used_data":14500,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540524,"statfs":{"total":1151940427776,"available":1151529496576,"internally_reserved":0,"allocated":14848000,"data_stored":8636782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988841"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589208+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:17:09.589208+0000","last_peered":"2024-04-23T18:17:09.589208+0000","last_clean":"2024-04-23T18:17:09.589208+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:17:09.589208+0000","last_undegraded":"2024-04-23T18:17:09.589208+0000","last_fullsized":"2024-04-23T18:17:09.589208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.868165+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:16:40.868165+0000","last_peered":"2024-04-23T18:16:40.868165+0000","last_clean":"2024-04-23T18:16:40.868165+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:16:40.868165+0000","last_undegraded":"2024-04-23T18:16:40.868165+0000","last_fullsized":"2024-04-23T18:16:40.868165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.872580+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:16:40.872580+0000","last_peered":"2024-04-23T18:16:40.872580+0000","last_clean":"2024-04-23T18:16:40.872580+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:16:40.872580+0000","last_undegraded":"2024-04-23T18:16:40.872580+0000","last_fullsized":"2024-04-23T18:16:40.872580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589665+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:17:09.589665+0000","last_peered":"2024-04-23T18:17:09.589665+0000","last_clean":"2024-04-23T18:17:09.589665+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:17:09.589665+0000","last_undegraded":"2024-04-23T18:17:09.589665+0000","last_fullsized":"2024-04-23T18:17:09.589665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537189+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:17:09.537189+0000","last_peered":"2024-04-23T18:17:09.537189+0000","last_clean":"2024-04-23T18:17:09.537189+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:17:09.537189+0000","last_undegraded":"2024-04-23T18:17:09.537189+0000","last_fullsized":"2024-04-23T18:17:09.537189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546509+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:17:09.546509+0000","last_peered":"2024-04-23T18:17:09.546509+0000","last_clean":"2024-04-23T18:17:09.546509+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:17:09.546509+0000","last_undegraded":"2024-04-23T18:17:09.546509+0000","last_fullsized":"2024-04-23T18:17:09.546509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442050+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:17:09.442050+0000","last_peered":"2024-04-23T18:17:09.442050+0000","last_clean":"2024-04-23T18:17:09.442050+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:17:09.442050+0000","last_undegraded":"2024-04-23T18:17:09.442050+0000","last_fullsized":"2024-04-23T18:17:09.442050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537380+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:17:09.537380+0000","last_peered":"2024-04-23T18:17:09.537380+0000","last_clean":"2024-04-23T18:17:09.537380+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:17:09.537380+0000","last_undegraded":"2024-04-23T18:17:09.537380+0000","last_fullsized":"2024-04-23T18:17:09.537380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589314+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:17:09.589314+0000","last_peered":"2024-04-23T18:17:09.589314+0000","last_clean":"2024-04-23T18:17:09.589314+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:17:09.589314+0000","last_undegraded":"2024-04-23T18:17:09.589314+0000","last_fullsized":"2024-04-23T18:17:09.589314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546802+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:17:09.546802+0000","last_peered":"2024-04-23T18:17:09.546802+0000","last_clean":"2024-04-23T18:17:09.546802+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:17:09.546802+0000","last_undegraded":"2024-04-23T18:17:09.546802+0000","last_fullsized":"2024-04-23T18:17:09.546802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442482+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:17:09.442482+0000","last_peered":"2024-04-23T18:17:09.442482+0000","last_clean":"2024-04-23T18:17:09.442482+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:17:09.442482+0000","last_undegraded":"2024-04-23T18:17:09.442482+0000","last_fullsized":"2024-04-23T18:17:09.442482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589542+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:17:09.589542+0000","last_peered":"2024-04-23T18:17:09.589542+0000","last_clean":"2024-04-23T18:17:09.589542+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:17:09.589542+0000","last_undegraded":"2024-04-23T18:17:09.589542+0000","last_fullsized":"2024-04-23T18:17:09.589542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589449+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:17:09.589449+0000","last_peered":"2024-04-23T18:17:09.589449+0000","last_clean":"2024-04-23T18:17:09.589449+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:17:09.589449+0000","last_undegraded":"2024-04-23T18:17:09.589449+0000","last_fullsized":"2024-04-23T18:17:09.589449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.869827+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:16:40.869827+0000","last_peered":"2024-04-23T18:16:40.869827+0000","last_clean":"2024-04-23T18:16:40.869827+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:16:40.869827+0000","last_undegraded":"2024-04-23T18:16:40.869827+0000","last_fullsized":"2024-04-23T18:16:40.869827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.862991+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:16:40.862991+0000","last_peered":"2024-04-23T18:16:40.862991+0000","last_clean":"2024-04-23T18:16:40.862991+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:16:40.862991+0000","last_undegraded":"2024-04-23T18:16:40.862991+0000","last_fullsized":"2024-04-23T18:16:40.862991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588754+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:17:09.588754+0000","last_peered":"2024-04-23T18:17:09.588754+0000","last_clean":"2024-04-23T18:17:09.588754+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:17:09.588754+0000","last_undegraded":"2024-04-23T18:17:09.588754+0000","last_fullsized":"2024-04-23T18:17:09.588754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589333+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:17:09.589333+0000","last_peered":"2024-04-23T18:17:09.589333+0000","last_clean":"2024-04-23T18:17:09.589333+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:17:09.589333+0000","last_undegraded":"2024-04-23T18:17:09.589333+0000","last_fullsized":"2024-04-23T18:17:09.589333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440816+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:17:09.440816+0000","last_peered":"2024-04-23T18:17:09.440816+0000","last_clean":"2024-04-23T18:17:09.440816+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:17:09.440816+0000","last_undegraded":"2024-04-23T18:17:09.440816+0000","last_fullsized":"2024-04-23T18:17:09.440816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442415+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:17:09.442415+0000","last_peered":"2024-04-23T18:17:09.442415+0000","last_clean":"2024-04-23T18:17:09.442415+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:17:09.442415+0000","last_undegraded":"2024-04-23T18:17:09.442415+0000","last_fullsized":"2024-04-23T18:17:09.442415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440896+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:17:09.440896+0000","last_peered":"2024-04-23T18:17:09.440896+0000","last_clean":"2024-04-23T18:17:09.440896+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:17:09.440896+0000","last_undegraded":"2024-04-23T18:17:09.440896+0000","last_fullsized":"2024-04-23T18:17:09.440896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538037+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:17:09.538037+0000","last_peered":"2024-04-23T18:17:09.538037+0000","last_clean":"2024-04-23T18:17:09.538037+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:17:09.538037+0000","last_undegraded":"2024-04-23T18:17:09.538037+0000","last_fullsized":"2024-04-23T18:17:09.538037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538172+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:17:09.538172+0000","last_peered":"2024-04-23T18:17:09.538172+0000","last_clean":"2024-04-23T18:17:09.538172+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:17:09.538172+0000","last_undegraded":"2024-04-23T18:17:09.538172+0000","last_fullsized":"2024-04-23T18:17:09.538172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.872485+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:16:40.872485+0000","last_peered":"2024-04-23T18:16:40.872485+0000","last_clean":"2024-04-23T18:16:40.872485+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:16:40.872485+0000","last_undegraded":"2024-04-23T18:16:40.872485+0000","last_fullsized":"2024-04-23T18:16:40.872485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537536+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:17:09.537536+0000","last_peered":"2024-04-23T18:17:09.537536+0000","last_clean":"2024-04-23T18:17:09.537536+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:17:09.537536+0000","last_undegraded":"2024-04-23T18:17:09.537536+0000","last_fullsized":"2024-04-23T18:17:09.537536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588909+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:17:09.588909+0000","last_peered":"2024-04-23T18:17:09.588909+0000","last_clean":"2024-04-23T18:17:09.588909+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:17:09.588909+0000","last_undegraded":"2024-04-23T18:17:09.588909+0000","last_fullsized":"2024-04-23T18:17:09.588909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537434+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:17:09.537434+0000","last_peered":"2024-04-23T18:17:09.537434+0000","last_clean":"2024-04-23T18:17:09.537434+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:17:09.537434+0000","last_undegraded":"2024-04-23T18:17:09.537434+0000","last_fullsized":"2024-04-23T18:17:09.537434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543969+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:17:09.543969+0000","last_peered":"2024-04-23T18:17:09.543969+0000","last_clean":"2024-04-23T18:17:09.543969+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:17:09.543969+0000","last_undegraded":"2024-04-23T18:17:09.543969+0000","last_fullsized":"2024-04-23T18:17:09.543969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":162,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386402+0000","last_change":"2024-04-23T18:16:12.309950+0000","last_active":"2024-04-23T18:17:10.386402+0000","last_peered":"2024-04-23T18:17:10.386402+0000","last_clean":"2024-04-23T18:17:10.386402+0000","last_became_active":"2024-04-23T18:16:12.305208+0000","last_became_peered":"2024-04-23T18:16:12.305208+0000","last_unstale":"2024-04-23T18:17:10.386402+0000","last_undegraded":"2024-04-23T18:17:10.386402+0000","last_fullsized":"2024-04-23T18:17:10.386402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:02.996009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537139+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:17:09.537139+0000","last_peered":"2024-04-23T18:17:09.537139+0000","last_clean":"2024-04-23T18:17:09.537139+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:17:09.537139+0000","last_undegraded":"2024-04-23T18:17:09.537139+0000","last_fullsized":"2024-04-23T18:17:09.537139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546457+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:17:09.546457+0000","last_peered":"2024-04-23T18:17:09.546457+0000","last_clean":"2024-04-23T18:17:09.546457+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:17:09.546457+0000","last_undegraded":"2024-04-23T18:17:09.546457+0000","last_fullsized":"2024-04-23T18:17:09.546457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543773+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:17:09.543773+0000","last_peered":"2024-04-23T18:17:09.543773+0000","last_clean":"2024-04-23T18:17:09.543773+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:17:09.543773+0000","last_undegraded":"2024-04-23T18:17:09.543773+0000","last_fullsized":"2024-04-23T18:17:09.543773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589412+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:17:09.589412+0000","last_peered":"2024-04-23T18:17:09.589412+0000","last_clean":"2024-04-23T18:17:09.589412+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:17:09.589412+0000","last_undegraded":"2024-04-23T18:17:09.589412+0000","last_fullsized":"2024-04-23T18:17:09.589412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537456+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:17:09.537456+0000","last_peered":"2024-04-23T18:17:09.537456+0000","last_clean":"2024-04-23T18:17:09.537456+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:17:09.537456+0000","last_undegraded":"2024-04-23T18:17:09.537456+0000","last_fullsized":"2024-04-23T18:17:09.537456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543566+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:17:09.543566+0000","last_peered":"2024-04-23T18:17:09.543566+0000","last_clean":"2024-04-23T18:17:09.543566+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:17:09.543566+0000","last_undegraded":"2024-04-23T18:17:09.543566+0000","last_fullsized":"2024-04-23T18:17:09.543566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440595+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:17:09.440595+0000","last_peered":"2024-04-23T18:17:09.440595+0000","last_clean":"2024-04-23T18:17:09.440595+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:17:09.440595+0000","last_undegraded":"2024-04-23T18:17:09.440595+0000","last_fullsized":"2024-04-23T18:17:09.440595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":144,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.692105+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:17:09.692105+0000","last_peered":"2024-04-23T18:17:09.692105+0000","last_clean":"2024-04-23T18:17:09.692105+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:17:09.692105+0000","last_undegraded":"2024-04-23T18:17:09.692105+0000","last_fullsized":"2024-04-23T18:17:09.692105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589181+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:17:09.589181+0000","last_peered":"2024-04-23T18:17:09.589181+0000","last_clean":"2024-04-23T18:17:09.589181+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:17:09.589181+0000","last_undegraded":"2024-04-23T18:17:09.589181+0000","last_fullsized":"2024-04-23T18:17:09.589181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.868443+0000","last_change":"2024-04-23T18:16:12.306548+0000","last_active":"2024-04-23T18:16:40.868443+0000","last_peered":"2024-04-23T18:16:40.868443+0000","last_clean":"2024-04-23T18:16:40.868443+0000","last_became_active":"2024-04-23T18:16:12.306227+0000","last_became_peered":"2024-04-23T18:16:12.306227+0000","last_unstale":"2024-04-23T18:16:40.868443+0000","last_undegraded":"2024-04-23T18:16:40.868443+0000","last_fullsized":"2024-04-23T18:16:40.868443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:49.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.544029+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:17:09.544029+0000","last_peered":"2024-04-23T18:17:09.544029+0000","last_clean":"2024-04-23T18:17:09.544029+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:17:09.544029+0000","last_undegraded":"2024-04-23T18:17:09.544029+0000","last_fullsized":"2024-04-23T18:17:09.544029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589084+0000","last_change":"2024-04-23T18:16:12.304182+0000","last_active":"2024-04-23T18:17:09.589084+0000","last_peered":"2024-04-23T18:17:09.589084+0000","last_clean":"2024-04-23T18:17:09.589084+0000","last_became_active":"2024-04-23T18:16:12.303708+0000","last_became_peered":"2024-04-23T18:16:12.303708+0000","last_unstale":"2024-04-23T18:17:09.589084+0000","last_undegraded":"2024-04-23T18:17:09.589084+0000","last_fullsized":"2024-04-23T18:17:09.589084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:32.224284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386516+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:17:10.386516+0000","last_peered":"2024-04-23T18:17:10.386516+0000","last_clean":"2024-04-23T18:17:10.386516+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:17:10.386516+0000","last_undegraded":"2024-04-23T18:17:10.386516+0000","last_fullsized":"2024-04-23T18:17:10.386516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":191,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537357+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:17:09.537357+0000","last_peered":"2024-04-23T18:17:09.537357+0000","last_clean":"2024-04-23T18:17:09.537357+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:17:09.537357+0000","last_undegraded":"2024-04-23T18:17:09.537357+0000","last_fullsized":"2024-04-23T18:17:09.537357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588956+0000","last_change":"2024-04-23T18:16:12.314402+0000","last_active":"2024-04-23T18:17:09.588956+0000","last_peered":"2024-04-23T18:17:09.588956+0000","last_clean":"2024-04-23T18:17:09.588956+0000","last_became_active":"2024-04-23T18:16:12.305588+0000","last_became_peered":"2024-04-23T18:16:12.305588+0000","last_unstale":"2024-04-23T18:17:09.588956+0000","last_undegraded":"2024-04-23T18:17:09.588956+0000","last_fullsized":"2024-04-23T18:17:09.588956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:15.413380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589319+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:17:09.589319+0000","last_peered":"2024-04-23T18:17:09.589319+0000","last_clean":"2024-04-23T18:17:09.589319+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:17:09.589319+0000","last_undegraded":"2024-04-23T18:17:09.589319+0000","last_fullsized":"2024-04-23T18:17:09.589319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543781+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:17:09.543781+0000","last_peered":"2024-04-23T18:17:09.543781+0000","last_clean":"2024-04-23T18:17:09.543781+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:17:09.543781+0000","last_undegraded":"2024-04-23T18:17:09.543781+0000","last_fullsized":"2024-04-23T18:17:09.543781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442441+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:17:09.442441+0000","last_peered":"2024-04-23T18:17:09.442441+0000","last_clean":"2024-04-23T18:17:09.442441+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:17:09.442441+0000","last_undegraded":"2024-04-23T18:17:09.442441+0000","last_fullsized":"2024-04-23T18:17:09.442441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'8","reported_seq":154,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.858716+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:17:09.858716+0000","last_peered":"2024-04-23T18:17:09.858716+0000","last_clean":"2024-04-23T18:17:09.858716+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:17:09.858716+0000","last_undegraded":"2024-04-23T18:17:09.858716+0000","last_fullsized":"2024-04-23T18:17:09.858716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.441991+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:17:09.441991+0000","last_peered":"2024-04-23T18:17:09.441991+0000","last_clean":"2024-04-23T18:17:09.441991+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:17:09.441991+0000","last_undegraded":"2024-04-23T18:17:09.441991+0000","last_fullsized":"2024-04-23T18:17:09.441991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543656+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:17:09.543656+0000","last_peered":"2024-04-23T18:17:09.543656+0000","last_clean":"2024-04-23T18:17:09.543656+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:17:09.543656+0000","last_undegraded":"2024-04-23T18:17:09.543656+0000","last_fullsized":"2024-04-23T18:17:09.543656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589193+0000","last_change":"2024-04-23T18:16:12.304948+0000","last_active":"2024-04-23T18:17:09.589193+0000","last_peered":"2024-04-23T18:17:09.589193+0000","last_clean":"2024-04-23T18:17:09.589193+0000","last_became_active":"2024-04-23T18:16:12.304541+0000","last_became_peered":"2024-04-23T18:16:12.304541+0000","last_unstale":"2024-04-23T18:17:09.589193+0000","last_undegraded":"2024-04-23T18:17:09.589193+0000","last_fullsized":"2024-04-23T18:17:09.589193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:47.148967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440865+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:17:09.440865+0000","last_peered":"2024-04-23T18:17:09.440865+0000","last_clean":"2024-04-23T18:17:09.440865+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:17:09.440865+0000","last_undegraded":"2024-04-23T18:17:09.440865+0000","last_fullsized":"2024-04-23T18:17:09.440865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546656+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:17:09.546656+0000","last_peered":"2024-04-23T18:17:09.546656+0000","last_clean":"2024-04-23T18:17:09.546656+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:17:09.546656+0000","last_undegraded":"2024-04-23T18:17:09.546656+0000","last_fullsized":"2024-04-23T18:17:09.546656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442224+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:17:09.442224+0000","last_peered":"2024-04-23T18:17:09.442224+0000","last_clean":"2024-04-23T18:17:09.442224+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:17:09.442224+0000","last_undegraded":"2024-04-23T18:17:09.442224+0000","last_fullsized":"2024-04-23T18:17:09.442224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546711+0000","last_change":"2024-04-23T18:16:12.304513+0000","last_active":"2024-04-23T18:17:09.546711+0000","last_peered":"2024-04-23T18:17:09.546711+0000","last_clean":"2024-04-23T18:17:09.546711+0000","last_became_active":"2024-04-23T18:16:12.304069+0000","last_became_peered":"2024-04-23T18:16:12.304069+0000","last_unstale":"2024-04-23T18:17:09.546711+0000","last_undegraded":"2024-04-23T18:17:09.546711+0000","last_fullsized":"2024-04-23T18:17:09.546711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:58.834388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543731+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:17:09.543731+0000","last_peered":"2024-04-23T18:17:09.543731+0000","last_clean":"2024-04-23T18:17:09.543731+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:17:09.543731+0000","last_undegraded":"2024-04-23T18:17:09.543731+0000","last_fullsized":"2024-04-23T18:17:09.543731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.857714+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:17:09.857714+0000","last_peered":"2024-04-23T18:17:09.857714+0000","last_clean":"2024-04-23T18:17:09.857714+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:17:09.857714+0000","last_undegraded":"2024-04-23T18:17:09.857714+0000","last_fullsized":"2024-04-23T18:17:09.857714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537763+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:17:09.537763+0000","last_peered":"2024-04-23T18:17:09.537763+0000","last_clean":"2024-04-23T18:17:09.537763+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:17:09.537763+0000","last_undegraded":"2024-04-23T18:17:09.537763+0000","last_fullsized":"2024-04-23T18:17:09.537763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589454+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:17:09.589454+0000","last_peered":"2024-04-23T18:17:09.589454+0000","last_clean":"2024-04-23T18:17:09.589454+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:17:09.589454+0000","last_undegraded":"2024-04-23T18:17:09.589454+0000","last_fullsized":"2024-04-23T18:17:09.589454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442087+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:17:09.442087+0000","last_peered":"2024-04-23T18:17:09.442087+0000","last_clean":"2024-04-23T18:17:09.442087+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:17:09.442087+0000","last_undegraded":"2024-04-23T18:17:09.442087+0000","last_fullsized":"2024-04-23T18:17:09.442087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589185+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:17:09.589185+0000","last_peered":"2024-04-23T18:17:09.589185+0000","last_clean":"2024-04-23T18:17:09.589185+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:17:09.589185+0000","last_undegraded":"2024-04-23T18:17:09.589185+0000","last_fullsized":"2024-04-23T18:17:09.589185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442419+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:17:09.442419+0000","last_peered":"2024-04-23T18:17:09.442419+0000","last_clean":"2024-04-23T18:17:09.442419+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:17:09.442419+0000","last_undegraded":"2024-04-23T18:17:09.442419+0000","last_fullsized":"2024-04-23T18:17:09.442419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589751+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:17:09.589751+0000","last_peered":"2024-04-23T18:17:09.589751+0000","last_clean":"2024-04-23T18:17:09.589751+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:17:09.589751+0000","last_undegraded":"2024-04-23T18:17:09.589751+0000","last_fullsized":"2024-04-23T18:17:09.589751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'44","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.652430+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:17:09.652430+0000","last_peered":"2024-04-23T18:17:09.652430+0000","last_clean":"2024-04-23T18:17:09.652430+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:17:09.652430+0000","last_undegraded":"2024-04-23T18:17:09.652430+0000","last_fullsized":"2024-04-23T18:17:09.652430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537883+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:17:09.537883+0000","last_peered":"2024-04-23T18:17:09.537883+0000","last_clean":"2024-04-23T18:17:09.537883+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:17:09.537883+0000","last_undegraded":"2024-04-23T18:17:09.537883+0000","last_fullsized":"2024-04-23T18:17:09.537883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589819+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:17:09.589819+0000","last_peered":"2024-04-23T18:17:09.589819+0000","last_clean":"2024-04-23T18:17:09.589819+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:17:09.589819+0000","last_undegraded":"2024-04-23T18:17:09.589819+0000","last_fullsized":"2024-04-23T18:17:09.589819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.867271+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:16:40.867271+0000","last_peered":"2024-04-23T18:16:40.867271+0000","last_clean":"2024-04-23T18:16:40.867271+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:16:40.867271+0000","last_undegraded":"2024-04-23T18:16:40.867271+0000","last_fullsized":"2024-04-23T18:16:40.867271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588772+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:17:09.588772+0000","last_peered":"2024-04-23T18:17:09.588772+0000","last_clean":"2024-04-23T18:17:09.588772+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:17:09.588772+0000","last_undegraded":"2024-04-23T18:17:09.588772+0000","last_fullsized":"2024-04-23T18:17:09.588772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537556+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:17:09.537556+0000","last_peered":"2024-04-23T18:17:09.537556+0000","last_clean":"2024-04-23T18:17:09.537556+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:17:09.537556+0000","last_undegraded":"2024-04-23T18:17:09.537556+0000","last_fullsized":"2024-04-23T18:17:09.537556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589049+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:17:09.589049+0000","last_peered":"2024-04-23T18:17:09.589049+0000","last_clean":"2024-04-23T18:17:09.589049+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:17:09.589049+0000","last_undegraded":"2024-04-23T18:17:09.589049+0000","last_fullsized":"2024-04-23T18:17:09.589049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442308+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:17:09.442308+0000","last_peered":"2024-04-23T18:17:09.442308+0000","last_clean":"2024-04-23T18:17:09.442308+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:17:09.442308+0000","last_undegraded":"2024-04-23T18:17:09.442308+0000","last_fullsized":"2024-04-23T18:17:09.442308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440818+0000","last_change":"2024-04-23T18:16:12.304138+0000","last_active":"2024-04-23T18:17:09.440818+0000","last_peered":"2024-04-23T18:17:09.440818+0000","last_clean":"2024-04-23T18:17:09.440818+0000","last_became_active":"2024-04-23T18:16:12.303760+0000","last_became_peered":"2024-04-23T18:16:12.303760+0000","last_unstale":"2024-04-23T18:17:09.440818+0000","last_undegraded":"2024-04-23T18:17:09.440818+0000","last_fullsized":"2024-04-23T18:17:09.440818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:10.049511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589223+0000","last_change":"2024-04-23T18:16:12.317810+0000","last_active":"2024-04-23T18:17:09.589223+0000","last_peered":"2024-04-23T18:17:09.589223+0000","last_clean":"2024-04-23T18:17:09.589223+0000","last_became_active":"2024-04-23T18:16:12.307064+0000","last_became_peered":"2024-04-23T18:16:12.307064+0000","last_unstale":"2024-04-23T18:17:09.589223+0000","last_undegraded":"2024-04-23T18:17:09.589223+0000","last_fullsized":"2024-04-23T18:17:09.589223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:06:41.038382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442209+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:17:09.442209+0000","last_peered":"2024-04-23T18:17:09.442209+0000","last_clean":"2024-04-23T18:17:09.442209+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:17:09.442209+0000","last_undegraded":"2024-04-23T18:17:09.442209+0000","last_fullsized":"2024-04-23T18:17:09.442209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442303+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:17:09.442303+0000","last_peered":"2024-04-23T18:17:09.442303+0000","last_clean":"2024-04-23T18:17:09.442303+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:17:09.442303+0000","last_undegraded":"2024-04-23T18:17:09.442303+0000","last_fullsized":"2024-04-23T18:17:09.442303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.650266+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:17:09.650266+0000","last_peered":"2024-04-23T18:17:09.650266+0000","last_clean":"2024-04-23T18:17:09.650266+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:17:09.650266+0000","last_undegraded":"2024-04-23T18:17:09.650266+0000","last_fullsized":"2024-04-23T18:17:09.650266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442273+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:17:09.442273+0000","last_peered":"2024-04-23T18:17:09.442273+0000","last_clean":"2024-04-23T18:17:09.442273+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:17:09.442273+0000","last_undegraded":"2024-04-23T18:17:09.442273+0000","last_fullsized":"2024-04-23T18:17:09.442273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589800+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:17:09.589800+0000","last_peered":"2024-04-23T18:17:09.589800+0000","last_clean":"2024-04-23T18:17:09.589800+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:17:09.589800+0000","last_undegraded":"2024-04-23T18:17:09.589800+0000","last_fullsized":"2024-04-23T18:17:09.589800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442682+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:17:09.442682+0000","last_peered":"2024-04-23T18:17:09.442682+0000","last_clean":"2024-04-23T18:17:09.442682+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:17:09.442682+0000","last_undegraded":"2024-04-23T18:17:09.442682+0000","last_fullsized":"2024-04-23T18:17:09.442682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589247+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:17:09.589247+0000","last_peered":"2024-04-23T18:17:09.589247+0000","last_clean":"2024-04-23T18:17:09.589247+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:17:09.589247+0000","last_undegraded":"2024-04-23T18:17:09.589247+0000","last_fullsized":"2024-04-23T18:17:09.589247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442275+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:17:09.442275+0000","last_peered":"2024-04-23T18:17:09.442275+0000","last_clean":"2024-04-23T18:17:09.442275+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:17:09.442275+0000","last_undegraded":"2024-04-23T18:17:09.442275+0000","last_fullsized":"2024-04-23T18:17:09.442275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442192+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:17:09.442192+0000","last_peered":"2024-04-23T18:17:09.442192+0000","last_clean":"2024-04-23T18:17:09.442192+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:17:09.442192+0000","last_undegraded":"2024-04-23T18:17:09.442192+0000","last_fullsized":"2024-04-23T18:17:09.442192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.867353+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:16:40.867353+0000","last_peered":"2024-04-23T18:16:40.867353+0000","last_clean":"2024-04-23T18:16:40.867353+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:16:40.867353+0000","last_undegraded":"2024-04-23T18:16:40.867353+0000","last_fullsized":"2024-04-23T18:16:40.867353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589753+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:17:09.589753+0000","last_peered":"2024-04-23T18:17:09.589753+0000","last_clean":"2024-04-23T18:17:09.589753+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:17:09.589753+0000","last_undegraded":"2024-04-23T18:17:09.589753+0000","last_fullsized":"2024-04-23T18:17:09.589753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589099+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:17:09.589099+0000","last_peered":"2024-04-23T18:17:09.589099+0000","last_clean":"2024-04-23T18:17:09.589099+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:17:09.589099+0000","last_undegraded":"2024-04-23T18:17:09.589099+0000","last_fullsized":"2024-04-23T18:17:09.589099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442534+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:17:09.442534+0000","last_peered":"2024-04-23T18:17:09.442534+0000","last_clean":"2024-04-23T18:17:09.442534+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:17:09.442534+0000","last_undegraded":"2024-04-23T18:17:09.442534+0000","last_fullsized":"2024-04-23T18:17:09.442534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537737+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:17:09.537737+0000","last_peered":"2024-04-23T18:17:09.537737+0000","last_clean":"2024-04-23T18:17:09.537737+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:17:09.537737+0000","last_undegraded":"2024-04-23T18:17:09.537737+0000","last_fullsized":"2024-04-23T18:17:09.537737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543635+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:17:09.543635+0000","last_peered":"2024-04-23T18:17:09.543635+0000","last_clean":"2024-04-23T18:17:09.543635+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:17:09.543635+0000","last_undegraded":"2024-04-23T18:17:09.543635+0000","last_fullsized":"2024-04-23T18:17:09.543635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537500+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:17:09.537500+0000","last_peered":"2024-04-23T18:17:09.537500+0000","last_clean":"2024-04-23T18:17:09.537500+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:17:09.537500+0000","last_undegraded":"2024-04-23T18:17:09.537500+0000","last_fullsized":"2024-04-23T18:17:09.537500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589280+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:17:09.589280+0000","last_peered":"2024-04-23T18:17:09.589280+0000","last_clean":"2024-04-23T18:17:09.589280+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:17:09.589280+0000","last_undegraded":"2024-04-23T18:17:09.589280+0000","last_fullsized":"2024-04-23T18:17:09.589280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589566+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:17:09.589566+0000","last_peered":"2024-04-23T18:17:09.589566+0000","last_clean":"2024-04-23T18:17:09.589566+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:17:09.589566+0000","last_undegraded":"2024-04-23T18:17:09.589566+0000","last_fullsized":"2024-04-23T18:17:09.589566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588942+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:17:09.588942+0000","last_peered":"2024-04-23T18:17:09.588942+0000","last_clean":"2024-04-23T18:17:09.588942+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:17:09.588942+0000","last_undegraded":"2024-04-23T18:17:09.588942+0000","last_fullsized":"2024-04-23T18:17:09.588942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598588,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":836,"num_read_kb":529,"num_write":286,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":656,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1795764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":469660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1114623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36688,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708464,"statfs":{"total":95995035648,"available":95957467136,"internally_reserved":0,"allocated":1380352,"data_stored":863078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704108,"statfs":{"total":95995035648,"available":95953006592,"internally_reserved":0,"allocated":1581056,"data_stored":1063725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":536802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40452,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704700,"statfs":{"total":95995035648,"available":95953612800,"internally_reserved":0,"allocated":978944,"data_stored":461595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":537496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":994378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40464,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704688,"statfs":{"total":95995035648,"available":95953600512,"internally_reserved":0,"allocated":987136,"data_stored":472714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712108,"statfs":{"total":95995035648,"available":95961198592,"internally_reserved":0,"allocated":1728512,"data_stored":1192605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:17:14.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:17:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/127271523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/870766346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:14.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:14.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.944 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/493618554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/127271523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: Updating smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/870766346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:17:14.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/493618554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 -- 172.21.15.26:0/1427439721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9078101ac0 msgr2=0x7f9078105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 --2- 172.21.15.26:0/1427439721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9078101ac0 0x7f9078105ab0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9068009900 tx=0x7f906802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 -- 172.21.15.26:0/1427439721 shutdown_connections 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 --2- 172.21.15.26:0/1427439721 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 0x7f907810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 --2- 172.21.15.26:0/1427439721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9078101ac0 0x7f9078105ab0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 --2- 172.21.15.26:0/1427439721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f90781014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 -- 172.21.15.26:0/1427439721 >> 172.21.15.26:0/1427439721 conn(0x7f9078077fb0 msgr2=0x7f90780783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 -- 172.21.15.26:0/1427439721 shutdown_connections 2024-04-23T18:17:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.025+0000 7f907e151640 1 -- 172.21.15.26:0/1427439721 wait complete. 2024-04-23T18:17:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 Processor -- start 2024-04-23T18:17:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 -- start start 2024-04-23T18:17:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f9078197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 0x7f9078197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f90781110b0 con 0x7f907819bf50 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9078110f30 con 0x7f9078101110 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f907e151640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9078111230 con 0x7f9078106190 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f9077fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f9077fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48956/0 (socket says 172.21.15.26:48956) 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f9077fff640 1 -- 172.21.15.26:0/2122862305 learned_addr learned my addr 172.21.15.26:0/2122862305 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.026+0000 7f90777fe640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f9078197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9076ffd640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 0x7f9078197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 -- 172.21.15.26:0/2122862305 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 msgr2=0x7f9078197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 0x7f9078197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 -- 172.21.15.26:0/2122862305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 msgr2=0x7f9078197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f9078197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 -- 172.21.15.26:0/2122862305 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907819ca40 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f90777fe640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f9078197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9077fff640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f907000e960 tx=0x7f907000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070004d20 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9070005020 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907819cd30 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700116d0 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.027+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90781a44a0 con 0x7f907819bf50 2024-04-23T18:17:15.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.028+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90780691c0 con 0x7f907819bf50 2024-04-23T18:17:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.029+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f90700051d0 con 0x7f907819bf50 2024-04-23T18:17:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.030+0000 7f9074ff9640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 0x7f90600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.030+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f90700d0b30 con 0x7f907819bf50 2024-04-23T18:17:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.030+0000 7f90777fe640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 0x7f90600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.030+0000 7f90777fe640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 0x7f90600a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f906c00abe0 tx=0x7f906c00a5c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.034+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9070016030 con 0x7f907819bf50 2024-04-23T18:17:15.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.216+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9078101ac0 con 0x7f90600a3db0 2024-04-23T18:17:15.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.224+0000 7f9074ff9640 1 -- 172.21.15.26:0/2122862305 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257734 (secure 0 0 0) 0x7f9078101ac0 con 0x7f90600a3db0 2024-04-23T18:17:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:15.222 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:17:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 msgr2=0x7f90600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 0x7f90600a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f906c00abe0 tx=0x7f906c00a5c0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 msgr2=0x7f907819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f907000e960 tx=0x7f907000ee20 comp rx=0 tx=0).stop 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 shutdown_connections 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90600a3db0 0x7f90600a6270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f907819bf50 0x7f907819c400 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9078106190 0x7f9078197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 --2- 172.21.15.26:0/2122862305 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9078101110 0x7f9078197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 >> 172.21.15.26:0/2122862305 conn(0x7f9078077fb0 msgr2=0x7f90780ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 shutdown_connections 2024-04-23T18:17:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.227+0000 7f907e151640 1 -- 172.21.15.26:0/2122862305 wait complete. 2024-04-23T18:17:15.232 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:17:13.453977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189190,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":972,"num_read_kb":648,"num_write":469,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181392,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401300,"kb_used_data":14500,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540524,"statfs":{"total":1151940427776,"available":1151529496576,"internally_reserved":0,"allocated":14848000,"data_stored":8636782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988841"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589208+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:17:09.589208+0000","last_peered":"2024-04-23T18:17:09.589208+0000","last_clean":"2024-04-23T18:17:09.589208+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:17:09.589208+0000","last_undegraded":"2024-04-23T18:17:09.589208+0000","last_fullsized":"2024-04-23T18:17:09.589208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.868165+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:16:40.868165+0000","last_peered":"2024-04-23T18:16:40.868165+0000","last_clean":"2024-04-23T18:16:40.868165+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:16:40.868165+0000","last_undegraded":"2024-04-23T18:16:40.868165+0000","last_fullsized":"2024-04-23T18:16:40.868165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.872580+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:16:40.872580+0000","last_peered":"2024-04-23T18:16:40.872580+0000","last_clean":"2024-04-23T18:16:40.872580+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:16:40.872580+0000","last_undegraded":"2024-04-23T18:16:40.872580+0000","last_fullsized":"2024-04-23T18:16:40.872580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589665+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:17:09.589665+0000","last_peered":"2024-04-23T18:17:09.589665+0000","last_clean":"2024-04-23T18:17:09.589665+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:17:09.589665+0000","last_undegraded":"2024-04-23T18:17:09.589665+0000","last_fullsized":"2024-04-23T18:17:09.589665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537189+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:17:09.537189+0000","last_peered":"2024-04-23T18:17:09.537189+0000","last_clean":"2024-04-23T18:17:09.537189+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:17:09.537189+0000","last_undegraded":"2024-04-23T18:17:09.537189+0000","last_fullsized":"2024-04-23T18:17:09.537189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546509+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:17:09.546509+0000","last_peered":"2024-04-23T18:17:09.546509+0000","last_clean":"2024-04-23T18:17:09.546509+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:17:09.546509+0000","last_undegraded":"2024-04-23T18:17:09.546509+0000","last_fullsized":"2024-04-23T18:17:09.546509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442050+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:17:09.442050+0000","last_peered":"2024-04-23T18:17:09.442050+0000","last_clean":"2024-04-23T18:17:09.442050+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:17:09.442050+0000","last_undegraded":"2024-04-23T18:17:09.442050+0000","last_fullsized":"2024-04-23T18:17:09.442050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537380+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:17:09.537380+0000","last_peered":"2024-04-23T18:17:09.537380+0000","last_clean":"2024-04-23T18:17:09.537380+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:17:09.537380+0000","last_undegraded":"2024-04-23T18:17:09.537380+0000","last_fullsized":"2024-04-23T18:17:09.537380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589314+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:17:09.589314+0000","last_peered":"2024-04-23T18:17:09.589314+0000","last_clean":"2024-04-23T18:17:09.589314+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:17:09.589314+0000","last_undegraded":"2024-04-23T18:17:09.589314+0000","last_fullsized":"2024-04-23T18:17:09.589314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546802+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:17:09.546802+0000","last_peered":"2024-04-23T18:17:09.546802+0000","last_clean":"2024-04-23T18:17:09.546802+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:17:09.546802+0000","last_undegraded":"2024-04-23T18:17:09.546802+0000","last_fullsized":"2024-04-23T18:17:09.546802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442482+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:17:09.442482+0000","last_peered":"2024-04-23T18:17:09.442482+0000","last_clean":"2024-04-23T18:17:09.442482+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:17:09.442482+0000","last_undegraded":"2024-04-23T18:17:09.442482+0000","last_fullsized":"2024-04-23T18:17:09.442482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589542+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:17:09.589542+0000","last_peered":"2024-04-23T18:17:09.589542+0000","last_clean":"2024-04-23T18:17:09.589542+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:17:09.589542+0000","last_undegraded":"2024-04-23T18:17:09.589542+0000","last_fullsized":"2024-04-23T18:17:09.589542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589449+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:17:09.589449+0000","last_peered":"2024-04-23T18:17:09.589449+0000","last_clean":"2024-04-23T18:17:09.589449+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:17:09.589449+0000","last_undegraded":"2024-04-23T18:17:09.589449+0000","last_fullsized":"2024-04-23T18:17:09.589449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.869827+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:16:40.869827+0000","last_peered":"2024-04-23T18:16:40.869827+0000","last_clean":"2024-04-23T18:16:40.869827+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:16:40.869827+0000","last_undegraded":"2024-04-23T18:16:40.869827+0000","last_fullsized":"2024-04-23T18:16:40.869827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.862991+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:16:40.862991+0000","last_peered":"2024-04-23T18:16:40.862991+0000","last_clean":"2024-04-23T18:16:40.862991+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:16:40.862991+0000","last_undegraded":"2024-04-23T18:16:40.862991+0000","last_fullsized":"2024-04-23T18:16:40.862991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588754+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:17:09.588754+0000","last_peered":"2024-04-23T18:17:09.588754+0000","last_clean":"2024-04-23T18:17:09.588754+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:17:09.588754+0000","last_undegraded":"2024-04-23T18:17:09.588754+0000","last_fullsized":"2024-04-23T18:17:09.588754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589333+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:17:09.589333+0000","last_peered":"2024-04-23T18:17:09.589333+0000","last_clean":"2024-04-23T18:17:09.589333+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:17:09.589333+0000","last_undegraded":"2024-04-23T18:17:09.589333+0000","last_fullsized":"2024-04-23T18:17:09.589333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440816+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:17:09.440816+0000","last_peered":"2024-04-23T18:17:09.440816+0000","last_clean":"2024-04-23T18:17:09.440816+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:17:09.440816+0000","last_undegraded":"2024-04-23T18:17:09.440816+0000","last_fullsized":"2024-04-23T18:17:09.440816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442415+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:17:09.442415+0000","last_peered":"2024-04-23T18:17:09.442415+0000","last_clean":"2024-04-23T18:17:09.442415+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:17:09.442415+0000","last_undegraded":"2024-04-23T18:17:09.442415+0000","last_fullsized":"2024-04-23T18:17:09.442415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440896+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:17:09.440896+0000","last_peered":"2024-04-23T18:17:09.440896+0000","last_clean":"2024-04-23T18:17:09.440896+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:17:09.440896+0000","last_undegraded":"2024-04-23T18:17:09.440896+0000","last_fullsized":"2024-04-23T18:17:09.440896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538037+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:17:09.538037+0000","last_peered":"2024-04-23T18:17:09.538037+0000","last_clean":"2024-04-23T18:17:09.538037+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:17:09.538037+0000","last_undegraded":"2024-04-23T18:17:09.538037+0000","last_fullsized":"2024-04-23T18:17:09.538037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538172+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:17:09.538172+0000","last_peered":"2024-04-23T18:17:09.538172+0000","last_clean":"2024-04-23T18:17:09.538172+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:17:09.538172+0000","last_undegraded":"2024-04-23T18:17:09.538172+0000","last_fullsized":"2024-04-23T18:17:09.538172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.872485+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:16:40.872485+0000","last_peered":"2024-04-23T18:16:40.872485+0000","last_clean":"2024-04-23T18:16:40.872485+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:16:40.872485+0000","last_undegraded":"2024-04-23T18:16:40.872485+0000","last_fullsized":"2024-04-23T18:16:40.872485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537536+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:17:09.537536+0000","last_peered":"2024-04-23T18:17:09.537536+0000","last_clean":"2024-04-23T18:17:09.537536+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:17:09.537536+0000","last_undegraded":"2024-04-23T18:17:09.537536+0000","last_fullsized":"2024-04-23T18:17:09.537536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588909+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:17:09.588909+0000","last_peered":"2024-04-23T18:17:09.588909+0000","last_clean":"2024-04-23T18:17:09.588909+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:17:09.588909+0000","last_undegraded":"2024-04-23T18:17:09.588909+0000","last_fullsized":"2024-04-23T18:17:09.588909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537434+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:17:09.537434+0000","last_peered":"2024-04-23T18:17:09.537434+0000","last_clean":"2024-04-23T18:17:09.537434+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:17:09.537434+0000","last_undegraded":"2024-04-23T18:17:09.537434+0000","last_fullsized":"2024-04-23T18:17:09.537434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543969+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:17:09.543969+0000","last_peered":"2024-04-23T18:17:09.543969+0000","last_clean":"2024-04-23T18:17:09.543969+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:17:09.543969+0000","last_undegraded":"2024-04-23T18:17:09.543969+0000","last_fullsized":"2024-04-23T18:17:09.543969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":162,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386402+0000","last_change":"2024-04-23T18:16:12.309950+0000","last_active":"2024-04-23T18:17:10.386402+0000","last_peered":"2024-04-23T18:17:10.386402+0000","last_clean":"2024-04-23T18:17:10.386402+0000","last_became_active":"2024-04-23T18:16:12.305208+0000","last_became_peered":"2024-04-23T18:16:12.305208+0000","last_unstale":"2024-04-23T18:17:10.386402+0000","last_undegraded":"2024-04-23T18:17:10.386402+0000","last_fullsized":"2024-04-23T18:17:10.386402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:02.996009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537139+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:17:09.537139+0000","last_peered":"2024-04-23T18:17:09.537139+0000","last_clean":"2024-04-23T18:17:09.537139+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:17:09.537139+0000","last_undegraded":"2024-04-23T18:17:09.537139+0000","last_fullsized":"2024-04-23T18:17:09.537139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546457+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:17:09.546457+0000","last_peered":"2024-04-23T18:17:09.546457+0000","last_clean":"2024-04-23T18:17:09.546457+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:17:09.546457+0000","last_undegraded":"2024-04-23T18:17:09.546457+0000","last_fullsized":"2024-04-23T18:17:09.546457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543773+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:17:09.543773+0000","last_peered":"2024-04-23T18:17:09.543773+0000","last_clean":"2024-04-23T18:17:09.543773+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:17:09.543773+0000","last_undegraded":"2024-04-23T18:17:09.543773+0000","last_fullsized":"2024-04-23T18:17:09.543773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589412+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:17:09.589412+0000","last_peered":"2024-04-23T18:17:09.589412+0000","last_clean":"2024-04-23T18:17:09.589412+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:17:09.589412+0000","last_undegraded":"2024-04-23T18:17:09.589412+0000","last_fullsized":"2024-04-23T18:17:09.589412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537456+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:17:09.537456+0000","last_peered":"2024-04-23T18:17:09.537456+0000","last_clean":"2024-04-23T18:17:09.537456+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:17:09.537456+0000","last_undegraded":"2024-04-23T18:17:09.537456+0000","last_fullsized":"2024-04-23T18:17:09.537456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543566+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:17:09.543566+0000","last_peered":"2024-04-23T18:17:09.543566+0000","last_clean":"2024-04-23T18:17:09.543566+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:17:09.543566+0000","last_undegraded":"2024-04-23T18:17:09.543566+0000","last_fullsized":"2024-04-23T18:17:09.543566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440595+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:17:09.440595+0000","last_peered":"2024-04-23T18:17:09.440595+0000","last_clean":"2024-04-23T18:17:09.440595+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:17:09.440595+0000","last_undegraded":"2024-04-23T18:17:09.440595+0000","last_fullsized":"2024-04-23T18:17:09.440595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":144,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.692105+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:17:09.692105+0000","last_peered":"2024-04-23T18:17:09.692105+0000","last_clean":"2024-04-23T18:17:09.692105+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:17:09.692105+0000","last_undegraded":"2024-04-23T18:17:09.692105+0000","last_fullsized":"2024-04-23T18:17:09.692105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589181+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:17:09.589181+0000","last_peered":"2024-04-23T18:17:09.589181+0000","last_clean":"2024-04-23T18:17:09.589181+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:17:09.589181+0000","last_undegraded":"2024-04-23T18:17:09.589181+0000","last_fullsized":"2024-04-23T18:17:09.589181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.868443+0000","last_change":"2024-04-23T18:16:12.306548+0000","last_active":"2024-04-23T18:16:40.868443+0000","last_peered":"2024-04-23T18:16:40.868443+0000","last_clean":"2024-04-23T18:16:40.868443+0000","last_became_active":"2024-04-23T18:16:12.306227+0000","last_became_peered":"2024-04-23T18:16:12.306227+0000","last_unstale":"2024-04-23T18:16:40.868443+0000","last_undegraded":"2024-04-23T18:16:40.868443+0000","last_fullsized":"2024-04-23T18:16:40.868443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:49.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.544029+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:17:09.544029+0000","last_peered":"2024-04-23T18:17:09.544029+0000","last_clean":"2024-04-23T18:17:09.544029+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:17:09.544029+0000","last_undegraded":"2024-04-23T18:17:09.544029+0000","last_fullsized":"2024-04-23T18:17:09.544029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589084+0000","last_change":"2024-04-23T18:16:12.304182+0000","last_active":"2024-04-23T18:17:09.589084+0000","last_peered":"2024-04-23T18:17:09.589084+0000","last_clean":"2024-04-23T18:17:09.589084+0000","last_became_active":"2024-04-23T18:16:12.303708+0000","last_became_peered":"2024-04-23T18:16:12.303708+0000","last_unstale":"2024-04-23T18:17:09.589084+0000","last_undegraded":"2024-04-23T18:17:09.589084+0000","last_fullsized":"2024-04-23T18:17:09.589084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:32.224284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386516+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:17:10.386516+0000","last_peered":"2024-04-23T18:17:10.386516+0000","last_clean":"2024-04-23T18:17:10.386516+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:17:10.386516+0000","last_undegraded":"2024-04-23T18:17:10.386516+0000","last_fullsized":"2024-04-23T18:17:10.386516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":191,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537357+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:17:09.537357+0000","last_peered":"2024-04-23T18:17:09.537357+0000","last_clean":"2024-04-23T18:17:09.537357+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:17:09.537357+0000","last_undegraded":"2024-04-23T18:17:09.537357+0000","last_fullsized":"2024-04-23T18:17:09.537357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588956+0000","last_change":"2024-04-23T18:16:12.314402+0000","last_active":"2024-04-23T18:17:09.588956+0000","last_peered":"2024-04-23T18:17:09.588956+0000","last_clean":"2024-04-23T18:17:09.588956+0000","last_became_active":"2024-04-23T18:16:12.305588+0000","last_became_peered":"2024-04-23T18:16:12.305588+0000","last_unstale":"2024-04-23T18:17:09.588956+0000","last_undegraded":"2024-04-23T18:17:09.588956+0000","last_fullsized":"2024-04-23T18:17:09.588956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:15.413380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589319+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:17:09.589319+0000","last_peered":"2024-04-23T18:17:09.589319+0000","last_clean":"2024-04-23T18:17:09.589319+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:17:09.589319+0000","last_undegraded":"2024-04-23T18:17:09.589319+0000","last_fullsized":"2024-04-23T18:17:09.589319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543781+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:17:09.543781+0000","last_peered":"2024-04-23T18:17:09.543781+0000","last_clean":"2024-04-23T18:17:09.543781+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:17:09.543781+0000","last_undegraded":"2024-04-23T18:17:09.543781+0000","last_fullsized":"2024-04-23T18:17:09.543781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442441+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:17:09.442441+0000","last_peered":"2024-04-23T18:17:09.442441+0000","last_clean":"2024-04-23T18:17:09.442441+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:17:09.442441+0000","last_undegraded":"2024-04-23T18:17:09.442441+0000","last_fullsized":"2024-04-23T18:17:09.442441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'8","reported_seq":154,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.858716+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:17:09.858716+0000","last_peered":"2024-04-23T18:17:09.858716+0000","last_clean":"2024-04-23T18:17:09.858716+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:17:09.858716+0000","last_undegraded":"2024-04-23T18:17:09.858716+0000","last_fullsized":"2024-04-23T18:17:09.858716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.441991+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:17:09.441991+0000","last_peered":"2024-04-23T18:17:09.441991+0000","last_clean":"2024-04-23T18:17:09.441991+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:17:09.441991+0000","last_undegraded":"2024-04-23T18:17:09.441991+0000","last_fullsized":"2024-04-23T18:17:09.441991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543656+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:17:09.543656+0000","last_peered":"2024-04-23T18:17:09.543656+0000","last_clean":"2024-04-23T18:17:09.543656+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:17:09.543656+0000","last_undegraded":"2024-04-23T18:17:09.543656+0000","last_fullsized":"2024-04-23T18:17:09.543656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589193+0000","last_change":"2024-04-23T18:16:12.304948+0000","last_active":"2024-04-23T18:17:09.589193+0000","last_peered":"2024-04-23T18:17:09.589193+0000","last_clean":"2024-04-23T18:17:09.589193+0000","last_became_active":"2024-04-23T18:16:12.304541+0000","last_became_peered":"2024-04-23T18:16:12.304541+0000","last_unstale":"2024-04-23T18:17:09.589193+0000","last_undegraded":"2024-04-23T18:17:09.589193+0000","last_fullsized":"2024-04-23T18:17:09.589193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:47.148967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440865+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:17:09.440865+0000","last_peered":"2024-04-23T18:17:09.440865+0000","last_clean":"2024-04-23T18:17:09.440865+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:17:09.440865+0000","last_undegraded":"2024-04-23T18:17:09.440865+0000","last_fullsized":"2024-04-23T18:17:09.440865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546656+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:17:09.546656+0000","last_peered":"2024-04-23T18:17:09.546656+0000","last_clean":"2024-04-23T18:17:09.546656+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:17:09.546656+0000","last_undegraded":"2024-04-23T18:17:09.546656+0000","last_fullsized":"2024-04-23T18:17:09.546656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442224+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:17:09.442224+0000","last_peered":"2024-04-23T18:17:09.442224+0000","last_clean":"2024-04-23T18:17:09.442224+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:17:09.442224+0000","last_undegraded":"2024-04-23T18:17:09.442224+0000","last_fullsized":"2024-04-23T18:17:09.442224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546711+0000","last_change":"2024-04-23T18:16:12.304513+0000","last_active":"2024-04-23T18:17:09.546711+0000","last_peered":"2024-04-23T18:17:09.546711+0000","last_clean":"2024-04-23T18:17:09.546711+0000","last_became_active":"2024-04-23T18:16:12.304069+0000","last_became_peered":"2024-04-23T18:16:12.304069+0000","last_unstale":"2024-04-23T18:17:09.546711+0000","last_undegraded":"2024-04-23T18:17:09.546711+0000","last_fullsized":"2024-04-23T18:17:09.546711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:58.834388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543731+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:17:09.543731+0000","last_peered":"2024-04-23T18:17:09.543731+0000","last_clean":"2024-04-23T18:17:09.543731+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:17:09.543731+0000","last_undegraded":"2024-04-23T18:17:09.543731+0000","last_fullsized":"2024-04-23T18:17:09.543731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.857714+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:17:09.857714+0000","last_peered":"2024-04-23T18:17:09.857714+0000","last_clean":"2024-04-23T18:17:09.857714+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:17:09.857714+0000","last_undegraded":"2024-04-23T18:17:09.857714+0000","last_fullsized":"2024-04-23T18:17:09.857714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537763+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:17:09.537763+0000","last_peered":"2024-04-23T18:17:09.537763+0000","last_clean":"2024-04-23T18:17:09.537763+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:17:09.537763+0000","last_undegraded":"2024-04-23T18:17:09.537763+0000","last_fullsized":"2024-04-23T18:17:09.537763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589454+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:17:09.589454+0000","last_peered":"2024-04-23T18:17:09.589454+0000","last_clean":"2024-04-23T18:17:09.589454+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:17:09.589454+0000","last_undegraded":"2024-04-23T18:17:09.589454+0000","last_fullsized":"2024-04-23T18:17:09.589454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442087+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:17:09.442087+0000","last_peered":"2024-04-23T18:17:09.442087+0000","last_clean":"2024-04-23T18:17:09.442087+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:17:09.442087+0000","last_undegraded":"2024-04-23T18:17:09.442087+0000","last_fullsized":"2024-04-23T18:17:09.442087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589185+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:17:09.589185+0000","last_peered":"2024-04-23T18:17:09.589185+0000","last_clean":"2024-04-23T18:17:09.589185+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:17:09.589185+0000","last_undegraded":"2024-04-23T18:17:09.589185+0000","last_fullsized":"2024-04-23T18:17:09.589185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442419+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:17:09.442419+0000","last_peered":"2024-04-23T18:17:09.442419+0000","last_clean":"2024-04-23T18:17:09.442419+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:17:09.442419+0000","last_undegraded":"2024-04-23T18:17:09.442419+0000","last_fullsized":"2024-04-23T18:17:09.442419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589751+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:17:09.589751+0000","last_peered":"2024-04-23T18:17:09.589751+0000","last_clean":"2024-04-23T18:17:09.589751+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:17:09.589751+0000","last_undegraded":"2024-04-23T18:17:09.589751+0000","last_fullsized":"2024-04-23T18:17:09.589751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'44","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.652430+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:17:09.652430+0000","last_peered":"2024-04-23T18:17:09.652430+0000","last_clean":"2024-04-23T18:17:09.652430+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:17:09.652430+0000","last_undegraded":"2024-04-23T18:17:09.652430+0000","last_fullsized":"2024-04-23T18:17:09.652430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537883+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:17:09.537883+0000","last_peered":"2024-04-23T18:17:09.537883+0000","last_clean":"2024-04-23T18:17:09.537883+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:17:09.537883+0000","last_undegraded":"2024-04-23T18:17:09.537883+0000","last_fullsized":"2024-04-23T18:17:09.537883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589819+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:17:09.589819+0000","last_peered":"2024-04-23T18:17:09.589819+0000","last_clean":"2024-04-23T18:17:09.589819+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:17:09.589819+0000","last_undegraded":"2024-04-23T18:17:09.589819+0000","last_fullsized":"2024-04-23T18:17:09.589819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.867271+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:16:40.867271+0000","last_peered":"2024-04-23T18:16:40.867271+0000","last_clean":"2024-04-23T18:16:40.867271+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:16:40.867271+0000","last_undegraded":"2024-04-23T18:16:40.867271+0000","last_fullsized":"2024-04-23T18:16:40.867271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588772+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:17:09.588772+0000","last_peered":"2024-04-23T18:17:09.588772+0000","last_clean":"2024-04-23T18:17:09.588772+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:17:09.588772+0000","last_undegraded":"2024-04-23T18:17:09.588772+0000","last_fullsized":"2024-04-23T18:17:09.588772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537556+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:17:09.537556+0000","last_peered":"2024-04-23T18:17:09.537556+0000","last_clean":"2024-04-23T18:17:09.537556+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:17:09.537556+0000","last_undegraded":"2024-04-23T18:17:09.537556+0000","last_fullsized":"2024-04-23T18:17:09.537556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589049+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:17:09.589049+0000","last_peered":"2024-04-23T18:17:09.589049+0000","last_clean":"2024-04-23T18:17:09.589049+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:17:09.589049+0000","last_undegraded":"2024-04-23T18:17:09.589049+0000","last_fullsized":"2024-04-23T18:17:09.589049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442308+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:17:09.442308+0000","last_peered":"2024-04-23T18:17:09.442308+0000","last_clean":"2024-04-23T18:17:09.442308+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:17:09.442308+0000","last_undegraded":"2024-04-23T18:17:09.442308+0000","last_fullsized":"2024-04-23T18:17:09.442308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440818+0000","last_change":"2024-04-23T18:16:12.304138+0000","last_active":"2024-04-23T18:17:09.440818+0000","last_peered":"2024-04-23T18:17:09.440818+0000","last_clean":"2024-04-23T18:17:09.440818+0000","last_became_active":"2024-04-23T18:16:12.303760+0000","last_became_peered":"2024-04-23T18:16:12.303760+0000","last_unstale":"2024-04-23T18:17:09.440818+0000","last_undegraded":"2024-04-23T18:17:09.440818+0000","last_fullsized":"2024-04-23T18:17:09.440818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:10.049511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589223+0000","last_change":"2024-04-23T18:16:12.317810+0000","last_active":"2024-04-23T18:17:09.589223+0000","last_peered":"2024-04-23T18:17:09.589223+0000","last_clean":"2024-04-23T18:17:09.589223+0000","last_became_active":"2024-04-23T18:16:12.307064+0000","last_became_peered":"2024-04-23T18:16:12.307064+0000","last_unstale":"2024-04-23T18:17:09.589223+0000","last_undegraded":"2024-04-23T18:17:09.589223+0000","last_fullsized":"2024-04-23T18:17:09.589223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:06:41.038382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442209+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:17:09.442209+0000","last_peered":"2024-04-23T18:17:09.442209+0000","last_clean":"2024-04-23T18:17:09.442209+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:17:09.442209+0000","last_undegraded":"2024-04-23T18:17:09.442209+0000","last_fullsized":"2024-04-23T18:17:09.442209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442303+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:17:09.442303+0000","last_peered":"2024-04-23T18:17:09.442303+0000","last_clean":"2024-04-23T18:17:09.442303+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:17:09.442303+0000","last_undegraded":"2024-04-23T18:17:09.442303+0000","last_fullsized":"2024-04-23T18:17:09.442303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.650266+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:17:09.650266+0000","last_peered":"2024-04-23T18:17:09.650266+0000","last_clean":"2024-04-23T18:17:09.650266+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:17:09.650266+0000","last_undegraded":"2024-04-23T18:17:09.650266+0000","last_fullsized":"2024-04-23T18:17:09.650266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442273+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:17:09.442273+0000","last_peered":"2024-04-23T18:17:09.442273+0000","last_clean":"2024-04-23T18:17:09.442273+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:17:09.442273+0000","last_undegraded":"2024-04-23T18:17:09.442273+0000","last_fullsized":"2024-04-23T18:17:09.442273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589800+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:17:09.589800+0000","last_peered":"2024-04-23T18:17:09.589800+0000","last_clean":"2024-04-23T18:17:09.589800+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:17:09.589800+0000","last_undegraded":"2024-04-23T18:17:09.589800+0000","last_fullsized":"2024-04-23T18:17:09.589800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442682+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:17:09.442682+0000","last_peered":"2024-04-23T18:17:09.442682+0000","last_clean":"2024-04-23T18:17:09.442682+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:17:09.442682+0000","last_undegraded":"2024-04-23T18:17:09.442682+0000","last_fullsized":"2024-04-23T18:17:09.442682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589247+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:17:09.589247+0000","last_peered":"2024-04-23T18:17:09.589247+0000","last_clean":"2024-04-23T18:17:09.589247+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:17:09.589247+0000","last_undegraded":"2024-04-23T18:17:09.589247+0000","last_fullsized":"2024-04-23T18:17:09.589247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442275+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:17:09.442275+0000","last_peered":"2024-04-23T18:17:09.442275+0000","last_clean":"2024-04-23T18:17:09.442275+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:17:09.442275+0000","last_undegraded":"2024-04-23T18:17:09.442275+0000","last_fullsized":"2024-04-23T18:17:09.442275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442192+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:17:09.442192+0000","last_peered":"2024-04-23T18:17:09.442192+0000","last_clean":"2024-04-23T18:17:09.442192+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:17:09.442192+0000","last_undegraded":"2024-04-23T18:17:09.442192+0000","last_fullsized":"2024-04-23T18:17:09.442192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T18:16:40.867353+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:16:40.867353+0000","last_peered":"2024-04-23T18:16:40.867353+0000","last_clean":"2024-04-23T18:16:40.867353+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:16:40.867353+0000","last_undegraded":"2024-04-23T18:16:40.867353+0000","last_fullsized":"2024-04-23T18:16:40.867353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589753+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:17:09.589753+0000","last_peered":"2024-04-23T18:17:09.589753+0000","last_clean":"2024-04-23T18:17:09.589753+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:17:09.589753+0000","last_undegraded":"2024-04-23T18:17:09.589753+0000","last_fullsized":"2024-04-23T18:17:09.589753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589099+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:17:09.589099+0000","last_peered":"2024-04-23T18:17:09.589099+0000","last_clean":"2024-04-23T18:17:09.589099+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:17:09.589099+0000","last_undegraded":"2024-04-23T18:17:09.589099+0000","last_fullsized":"2024-04-23T18:17:09.589099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442534+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:17:09.442534+0000","last_peered":"2024-04-23T18:17:09.442534+0000","last_clean":"2024-04-23T18:17:09.442534+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:17:09.442534+0000","last_undegraded":"2024-04-23T18:17:09.442534+0000","last_fullsized":"2024-04-23T18:17:09.442534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537737+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:17:09.537737+0000","last_peered":"2024-04-23T18:17:09.537737+0000","last_clean":"2024-04-23T18:17:09.537737+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:17:09.537737+0000","last_undegraded":"2024-04-23T18:17:09.537737+0000","last_fullsized":"2024-04-23T18:17:09.537737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543635+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:17:09.543635+0000","last_peered":"2024-04-23T18:17:09.543635+0000","last_clean":"2024-04-23T18:17:09.543635+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:17:09.543635+0000","last_undegraded":"2024-04-23T18:17:09.543635+0000","last_fullsized":"2024-04-23T18:17:09.543635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537500+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:17:09.537500+0000","last_peered":"2024-04-23T18:17:09.537500+0000","last_clean":"2024-04-23T18:17:09.537500+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:17:09.537500+0000","last_undegraded":"2024-04-23T18:17:09.537500+0000","last_fullsized":"2024-04-23T18:17:09.537500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589280+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:17:09.589280+0000","last_peered":"2024-04-23T18:17:09.589280+0000","last_clean":"2024-04-23T18:17:09.589280+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:17:09.589280+0000","last_undegraded":"2024-04-23T18:17:09.589280+0000","last_fullsized":"2024-04-23T18:17:09.589280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589566+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:17:09.589566+0000","last_peered":"2024-04-23T18:17:09.589566+0000","last_clean":"2024-04-23T18:17:09.589566+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:17:09.589566+0000","last_undegraded":"2024-04-23T18:17:09.589566+0000","last_fullsized":"2024-04-23T18:17:09.589566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588942+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:17:09.588942+0000","last_peered":"2024-04-23T18:17:09.588942+0000","last_clean":"2024-04-23T18:17:09.588942+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:17:09.588942+0000","last_undegraded":"2024-04-23T18:17:09.588942+0000","last_fullsized":"2024-04-23T18:17:09.588942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598588,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":836,"num_read_kb":529,"num_write":286,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":656,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1795764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":469660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1114623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36688,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708464,"statfs":{"total":95995035648,"available":95957467136,"internally_reserved":0,"allocated":1380352,"data_stored":863078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704108,"statfs":{"total":95995035648,"available":95953006592,"internally_reserved":0,"allocated":1581056,"data_stored":1063725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":536802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40452,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704700,"statfs":{"total":95995035648,"available":95953612800,"internally_reserved":0,"allocated":978944,"data_stored":461595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":537496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":994378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40464,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704688,"statfs":{"total":95995035648,"available":95953600512,"internally_reserved":0,"allocated":987136,"data_stored":472714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712108,"statfs":{"total":95995035648,"available":95961198592,"internally_reserved":0,"allocated":1728512,"data_stored":1192605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:17:15.236 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-23T18:17:15.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:17:15.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.402+0000 7ff87da31640 1 -- 172.21.15.26:0/2765046932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8781010a0 msgr2=0x7ff878101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.402+0000 7ff87da31640 1 --2- 172.21.15.26:0/2765046932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8781010a0 0x7ff878101480 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff868009900 tx=0x7ff86802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:15.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 -- 172.21.15.26:0/2765046932 shutdown_connections 2024-04-23T18:17:15.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 --2- 172.21.15.26:0/2765046932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff878106170 0x7ff87810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 --2- 172.21.15.26:0/2765046932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878101a50 0x7ff878105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 --2- 172.21.15.26:0/2765046932 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff8781010a0 0x7ff878101480 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 -- 172.21.15.26:0/2765046932 >> 172.21.15.26:0/2765046932 conn(0x7ff878077f50 msgr2=0x7ff878078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 -- 172.21.15.26:0/2765046932 shutdown_connections 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 -- 172.21.15.26:0/2765046932 wait complete. 2024-04-23T18:17:15.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 Processor -- start 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 -- start start 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8781010a0 0x7ff878197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.403+0000 7ff87da31640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878106170 0x7ff878197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff8781110d0 con 0x7ff87819bef0 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff878110f50 con 0x7ff8781010a0 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff878111250 con 0x7ff878106170 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48972/0 (socket says 172.21.15.26:48972) 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 -- 172.21.15.26:0/1345590904 learned_addr learned my addr 172.21.15.26:0/1345590904 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff876ffd640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8781010a0 0x7ff878197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8767fc640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878106170 0x7ff878197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 -- 172.21.15.26:0/1345590904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878106170 msgr2=0x7ff878197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878106170 0x7ff878197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 -- 172.21.15.26:0/1345590904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8781010a0 msgr2=0x7ff878197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8781010a0 0x7ff878197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 -- 172.21.15.26:0/1345590904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87819c9e0 con 0x7ff87819bef0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff8777fe640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff87000eb10 tx=0x7ff87000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870004b20 con 0x7ff87819bef0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff870004cc0 con 0x7ff87819bef0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870010670 con 0x7ff87819bef0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87819ccd0 con 0x7ff87819bef0 2024-04-23T18:17:15.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.404+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8780693b0 con 0x7ff87819bef0 2024-04-23T18:17:15.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.406+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff878068d70 con 0x7ff87819bef0 2024-04-23T18:17:15.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.407+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7ff870005020 con 0x7ff87819bef0 2024-04-23T18:17:15.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.407+0000 7ff87ca2f640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 0x7ff8600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.407+0000 7ff876ffd640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 0x7ff8600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.407+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff8700d0e70 con 0x7ff87819bef0 2024-04-23T18:17:15.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.408+0000 7ff876ffd640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 0x7ff8600a6270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff868031440 tx=0x7ff868005b50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.412+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff87009d3d0 con 0x7ff87819bef0 2024-04-23T18:17:15.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.599+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff878069f00 con 0x7ff8600a3db0 2024-04-23T18:17:15.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.606+0000 7ff87ca2f640 1 -- 172.21.15.26:0/1345590904 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257748 (secure 0 0 0) 0x7ff878069f00 con 0x7ff8600a3db0 2024-04-23T18:17:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:15.604 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 msgr2=0x7ff8600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 0x7ff8600a6270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff868031440 tx=0x7ff868005b50 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 msgr2=0x7ff87819c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff87000eb10 tx=0x7ff87000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 shutdown_connections 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff8600a3db0 0x7ff8600a6270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff87819bef0 0x7ff87819c3a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff878106170 0x7ff878197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 --2- 172.21.15.26:0/1345590904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8781010a0 0x7ff878197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 >> 172.21.15.26:0/1345590904 conn(0x7ff878077f50 msgr2=0x7ff8780ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 shutdown_connections 2024-04-23T18:17:15.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.609+0000 7ff87da31640 1 -- 172.21.15.26:0/1345590904 wait complete. 2024-04-23T18:17:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-23T18:17:15.454295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189190,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1202,"num_read_kb":855,"num_write":568,"num_write_kb":4825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181392,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":918,"ondisk_log_size":918,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401316,"kb_used_data":14516,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540508,"statfs":{"total":1151940427776,"available":1151529480192,"internally_reserved":0,"allocated":14864384,"data_stored":8645709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":395752762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":230,"num_read_kb":207,"num_write":99,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.989159"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589208+0000","last_change":"2024-04-23T18:15:06.448864+0000","last_active":"2024-04-23T18:17:09.589208+0000","last_peered":"2024-04-23T18:17:09.589208+0000","last_clean":"2024-04-23T18:17:09.589208+0000","last_became_active":"2024-04-23T18:15:06.448548+0000","last_became_peered":"2024-04-23T18:15:06.448548+0000","last_unstale":"2024-04-23T18:17:09.589208+0000","last_undegraded":"2024-04-23T18:17:09.589208+0000","last_fullsized":"2024-04-23T18:17:09.589208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:46.451659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.608983+0000","last_change":"2024-04-23T18:15:06.446342+0000","last_active":"2024-04-23T18:17:10.608983+0000","last_peered":"2024-04-23T18:17:10.608983+0000","last_clean":"2024-04-23T18:17:10.608983+0000","last_became_active":"2024-04-23T18:15:06.445896+0000","last_became_peered":"2024-04-23T18:15:06.445896+0000","last_unstale":"2024-04-23T18:17:10.608983+0000","last_undegraded":"2024-04-23T18:17:10.608983+0000","last_fullsized":"2024-04-23T18:17:10.608983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:25.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.608814+0000","last_change":"2024-04-23T18:15:06.454322+0000","last_active":"2024-04-23T18:17:10.608814+0000","last_peered":"2024-04-23T18:17:10.608814+0000","last_clean":"2024-04-23T18:17:10.608814+0000","last_became_active":"2024-04-23T18:15:06.453949+0000","last_became_peered":"2024-04-23T18:15:06.453949+0000","last_unstale":"2024-04-23T18:17:10.608814+0000","last_undegraded":"2024-04-23T18:17:10.608814+0000","last_fullsized":"2024-04-23T18:17:10.608814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:59.221362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589665+0000","last_change":"2024-04-23T18:15:06.448207+0000","last_active":"2024-04-23T18:17:09.589665+0000","last_peered":"2024-04-23T18:17:09.589665+0000","last_clean":"2024-04-23T18:17:09.589665+0000","last_became_active":"2024-04-23T18:15:06.447363+0000","last_became_peered":"2024-04-23T18:15:06.447363+0000","last_unstale":"2024-04-23T18:17:09.589665+0000","last_undegraded":"2024-04-23T18:17:09.589665+0000","last_fullsized":"2024-04-23T18:17:09.589665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:55.435569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537189+0000","last_change":"2024-04-23T18:15:06.441546+0000","last_active":"2024-04-23T18:17:09.537189+0000","last_peered":"2024-04-23T18:17:09.537189+0000","last_clean":"2024-04-23T18:17:09.537189+0000","last_became_active":"2024-04-23T18:15:06.441201+0000","last_became_peered":"2024-04-23T18:15:06.441201+0000","last_unstale":"2024-04-23T18:17:09.537189+0000","last_undegraded":"2024-04-23T18:17:09.537189+0000","last_fullsized":"2024-04-23T18:17:09.537189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:28:00.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546509+0000","last_change":"2024-04-23T18:15:06.451115+0000","last_active":"2024-04-23T18:17:09.546509+0000","last_peered":"2024-04-23T18:17:09.546509+0000","last_clean":"2024-04-23T18:17:09.546509+0000","last_became_active":"2024-04-23T18:15:06.450762+0000","last_became_peered":"2024-04-23T18:15:06.450762+0000","last_unstale":"2024-04-23T18:17:09.546509+0000","last_undegraded":"2024-04-23T18:17:09.546509+0000","last_fullsized":"2024-04-23T18:17:09.546509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:07:27.523366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442050+0000","last_change":"2024-04-23T18:15:06.444906+0000","last_active":"2024-04-23T18:17:09.442050+0000","last_peered":"2024-04-23T18:17:09.442050+0000","last_clean":"2024-04-23T18:17:09.442050+0000","last_became_active":"2024-04-23T18:15:06.444591+0000","last_became_peered":"2024-04-23T18:15:06.444591+0000","last_unstale":"2024-04-23T18:17:09.442050+0000","last_undegraded":"2024-04-23T18:17:09.442050+0000","last_fullsized":"2024-04-23T18:17:09.442050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:32:44.875763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537380+0000","last_change":"2024-04-23T18:15:06.448906+0000","last_active":"2024-04-23T18:17:09.537380+0000","last_peered":"2024-04-23T18:17:09.537380+0000","last_clean":"2024-04-23T18:17:09.537380+0000","last_became_active":"2024-04-23T18:15:06.448492+0000","last_became_peered":"2024-04-23T18:15:06.448492+0000","last_unstale":"2024-04-23T18:17:09.537380+0000","last_undegraded":"2024-04-23T18:17:09.537380+0000","last_fullsized":"2024-04-23T18:17:09.537380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:11.342310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589314+0000","last_change":"2024-04-23T18:15:06.443719+0000","last_active":"2024-04-23T18:17:09.589314+0000","last_peered":"2024-04-23T18:17:09.589314+0000","last_clean":"2024-04-23T18:17:09.589314+0000","last_became_active":"2024-04-23T18:15:06.443333+0000","last_became_peered":"2024-04-23T18:15:06.443333+0000","last_unstale":"2024-04-23T18:17:09.589314+0000","last_undegraded":"2024-04-23T18:17:09.589314+0000","last_fullsized":"2024-04-23T18:17:09.589314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:40.035538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546802+0000","last_change":"2024-04-23T18:15:06.453462+0000","last_active":"2024-04-23T18:17:09.546802+0000","last_peered":"2024-04-23T18:17:09.546802+0000","last_clean":"2024-04-23T18:17:09.546802+0000","last_became_active":"2024-04-23T18:15:06.451683+0000","last_became_peered":"2024-04-23T18:15:06.451683+0000","last_unstale":"2024-04-23T18:17:09.546802+0000","last_undegraded":"2024-04-23T18:17:09.546802+0000","last_fullsized":"2024-04-23T18:17:09.546802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:42.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442482+0000","last_change":"2024-04-23T18:15:06.450090+0000","last_active":"2024-04-23T18:17:09.442482+0000","last_peered":"2024-04-23T18:17:09.442482+0000","last_clean":"2024-04-23T18:17:09.442482+0000","last_became_active":"2024-04-23T18:15:06.449745+0000","last_became_peered":"2024-04-23T18:15:06.449745+0000","last_unstale":"2024-04-23T18:17:09.442482+0000","last_undegraded":"2024-04-23T18:17:09.442482+0000","last_fullsized":"2024-04-23T18:17:09.442482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:46.527579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589542+0000","last_change":"2024-04-23T18:15:06.448981+0000","last_active":"2024-04-23T18:17:09.589542+0000","last_peered":"2024-04-23T18:17:09.589542+0000","last_clean":"2024-04-23T18:17:09.589542+0000","last_became_active":"2024-04-23T18:15:06.448597+0000","last_became_peered":"2024-04-23T18:15:06.448597+0000","last_unstale":"2024-04-23T18:17:09.589542+0000","last_undegraded":"2024-04-23T18:17:09.589542+0000","last_fullsized":"2024-04-23T18:17:09.589542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:08.739081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589449+0000","last_change":"2024-04-23T18:15:06.443731+0000","last_active":"2024-04-23T18:17:09.589449+0000","last_peered":"2024-04-23T18:17:09.589449+0000","last_clean":"2024-04-23T18:17:09.589449+0000","last_became_active":"2024-04-23T18:15:06.443409+0000","last_became_peered":"2024-04-23T18:15:06.443409+0000","last_unstale":"2024-04-23T18:17:09.589449+0000","last_undegraded":"2024-04-23T18:17:09.589449+0000","last_fullsized":"2024-04-23T18:17:09.589449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:52.743730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.608498+0000","last_change":"2024-04-23T18:15:06.447174+0000","last_active":"2024-04-23T18:17:10.608498+0000","last_peered":"2024-04-23T18:17:10.608498+0000","last_clean":"2024-04-23T18:17:10.608498+0000","last_became_active":"2024-04-23T18:15:06.446655+0000","last_became_peered":"2024-04-23T18:15:06.446655+0000","last_unstale":"2024-04-23T18:17:10.608498+0000","last_undegraded":"2024-04-23T18:17:10.608498+0000","last_fullsized":"2024-04-23T18:17:10.608498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:39.510205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.608470+0000","last_change":"2024-04-23T18:15:06.451983+0000","last_active":"2024-04-23T18:17:10.608470+0000","last_peered":"2024-04-23T18:17:10.608470+0000","last_clean":"2024-04-23T18:17:10.608470+0000","last_became_active":"2024-04-23T18:15:06.451208+0000","last_became_peered":"2024-04-23T18:15:06.451208+0000","last_unstale":"2024-04-23T18:17:10.608470+0000","last_undegraded":"2024-04-23T18:17:10.608470+0000","last_fullsized":"2024-04-23T18:17:10.608470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:58:44.672101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588754+0000","last_change":"2024-04-23T18:15:06.450068+0000","last_active":"2024-04-23T18:17:09.588754+0000","last_peered":"2024-04-23T18:17:09.588754+0000","last_clean":"2024-04-23T18:17:09.588754+0000","last_became_active":"2024-04-23T18:15:06.449759+0000","last_became_peered":"2024-04-23T18:15:06.449759+0000","last_unstale":"2024-04-23T18:17:09.588754+0000","last_undegraded":"2024-04-23T18:17:09.588754+0000","last_fullsized":"2024-04-23T18:17:09.588754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:16.781494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589333+0000","last_change":"2024-04-23T18:15:06.449763+0000","last_active":"2024-04-23T18:17:09.589333+0000","last_peered":"2024-04-23T18:17:09.589333+0000","last_clean":"2024-04-23T18:17:09.589333+0000","last_became_active":"2024-04-23T18:15:06.449370+0000","last_became_peered":"2024-04-23T18:15:06.449370+0000","last_unstale":"2024-04-23T18:17:09.589333+0000","last_undegraded":"2024-04-23T18:17:09.589333+0000","last_fullsized":"2024-04-23T18:17:09.589333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:24:04.950453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440816+0000","last_change":"2024-04-23T18:15:06.442516+0000","last_active":"2024-04-23T18:17:09.440816+0000","last_peered":"2024-04-23T18:17:09.440816+0000","last_clean":"2024-04-23T18:17:09.440816+0000","last_became_active":"2024-04-23T18:15:06.442160+0000","last_became_peered":"2024-04-23T18:15:06.442160+0000","last_unstale":"2024-04-23T18:17:09.440816+0000","last_undegraded":"2024-04-23T18:17:09.440816+0000","last_fullsized":"2024-04-23T18:17:09.440816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:48:00.492922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442415+0000","last_change":"2024-04-23T18:15:06.444783+0000","last_active":"2024-04-23T18:17:09.442415+0000","last_peered":"2024-04-23T18:17:09.442415+0000","last_clean":"2024-04-23T18:17:09.442415+0000","last_became_active":"2024-04-23T18:15:06.442372+0000","last_became_peered":"2024-04-23T18:15:06.442372+0000","last_unstale":"2024-04-23T18:17:09.442415+0000","last_undegraded":"2024-04-23T18:17:09.442415+0000","last_fullsized":"2024-04-23T18:17:09.442415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:44:12.452110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440896+0000","last_change":"2024-04-23T18:15:05.690304+0000","last_active":"2024-04-23T18:17:09.440896+0000","last_peered":"2024-04-23T18:17:09.440896+0000","last_clean":"2024-04-23T18:17:09.440896+0000","last_became_active":"2024-04-23T18:15:05.689954+0000","last_became_peered":"2024-04-23T18:15:05.689954+0000","last_unstale":"2024-04-23T18:17:09.440896+0000","last_undegraded":"2024-04-23T18:17:09.440896+0000","last_fullsized":"2024-04-23T18:17:09.440896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:20.735140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538037+0000","last_change":"2024-04-23T18:15:06.447629+0000","last_active":"2024-04-23T18:17:09.538037+0000","last_peered":"2024-04-23T18:17:09.538037+0000","last_clean":"2024-04-23T18:17:09.538037+0000","last_became_active":"2024-04-23T18:15:06.447257+0000","last_became_peered":"2024-04-23T18:15:06.447257+0000","last_unstale":"2024-04-23T18:17:09.538037+0000","last_undegraded":"2024-04-23T18:17:09.538037+0000","last_fullsized":"2024-04-23T18:17:09.538037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.538172+0000","last_change":"2024-04-23T18:15:05.690173+0000","last_active":"2024-04-23T18:17:09.538172+0000","last_peered":"2024-04-23T18:17:09.538172+0000","last_clean":"2024-04-23T18:17:09.538172+0000","last_became_active":"2024-04-23T18:15:05.687179+0000","last_became_peered":"2024-04-23T18:15:05.687179+0000","last_unstale":"2024-04-23T18:17:09.538172+0000","last_undegraded":"2024-04-23T18:17:09.538172+0000","last_fullsized":"2024-04-23T18:17:09.538172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:33:02.397951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.609085+0000","last_change":"2024-04-23T18:15:06.448309+0000","last_active":"2024-04-23T18:17:10.609085+0000","last_peered":"2024-04-23T18:17:10.609085+0000","last_clean":"2024-04-23T18:17:10.609085+0000","last_became_active":"2024-04-23T18:15:06.447996+0000","last_became_peered":"2024-04-23T18:15:06.447996+0000","last_unstale":"2024-04-23T18:17:10.609085+0000","last_undegraded":"2024-04-23T18:17:10.609085+0000","last_fullsized":"2024-04-23T18:17:10.609085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:07:03.751039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537536+0000","last_change":"2024-04-23T18:15:05.695862+0000","last_active":"2024-04-23T18:17:09.537536+0000","last_peered":"2024-04-23T18:17:09.537536+0000","last_clean":"2024-04-23T18:17:09.537536+0000","last_became_active":"2024-04-23T18:15:05.695575+0000","last_became_peered":"2024-04-23T18:15:05.695575+0000","last_unstale":"2024-04-23T18:17:09.537536+0000","last_undegraded":"2024-04-23T18:17:09.537536+0000","last_fullsized":"2024-04-23T18:17:09.537536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:38.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588909+0000","last_change":"2024-04-23T18:15:06.446834+0000","last_active":"2024-04-23T18:17:09.588909+0000","last_peered":"2024-04-23T18:17:09.588909+0000","last_clean":"2024-04-23T18:17:09.588909+0000","last_became_active":"2024-04-23T18:15:06.446510+0000","last_became_peered":"2024-04-23T18:15:06.446510+0000","last_unstale":"2024-04-23T18:17:09.588909+0000","last_undegraded":"2024-04-23T18:17:09.588909+0000","last_fullsized":"2024-04-23T18:17:09.588909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:52.059090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537434+0000","last_change":"2024-04-23T18:15:05.687637+0000","last_active":"2024-04-23T18:17:09.537434+0000","last_peered":"2024-04-23T18:17:09.537434+0000","last_clean":"2024-04-23T18:17:09.537434+0000","last_became_active":"2024-04-23T18:15:05.687281+0000","last_became_peered":"2024-04-23T18:15:05.687281+0000","last_unstale":"2024-04-23T18:17:09.537434+0000","last_undegraded":"2024-04-23T18:17:09.537434+0000","last_fullsized":"2024-04-23T18:17:09.537434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:02:34.768518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543969+0000","last_change":"2024-04-23T18:15:06.450396+0000","last_active":"2024-04-23T18:17:09.543969+0000","last_peered":"2024-04-23T18:17:09.543969+0000","last_clean":"2024-04-23T18:17:09.543969+0000","last_became_active":"2024-04-23T18:15:06.449459+0000","last_became_peered":"2024-04-23T18:15:06.449459+0000","last_unstale":"2024-04-23T18:17:09.543969+0000","last_undegraded":"2024-04-23T18:17:09.543969+0000","last_fullsized":"2024-04-23T18:17:09.543969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:15.731657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":162,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386402+0000","last_change":"2024-04-23T18:16:12.309950+0000","last_active":"2024-04-23T18:17:10.386402+0000","last_peered":"2024-04-23T18:17:10.386402+0000","last_clean":"2024-04-23T18:17:10.386402+0000","last_became_active":"2024-04-23T18:16:12.305208+0000","last_became_peered":"2024-04-23T18:16:12.305208+0000","last_unstale":"2024-04-23T18:17:10.386402+0000","last_undegraded":"2024-04-23T18:17:10.386402+0000","last_fullsized":"2024-04-23T18:17:10.386402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:02.996009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537139+0000","last_change":"2024-04-23T18:15:06.445443+0000","last_active":"2024-04-23T18:17:09.537139+0000","last_peered":"2024-04-23T18:17:09.537139+0000","last_clean":"2024-04-23T18:17:09.537139+0000","last_became_active":"2024-04-23T18:15:06.444984+0000","last_became_peered":"2024-04-23T18:15:06.444984+0000","last_unstale":"2024-04-23T18:17:09.537139+0000","last_undegraded":"2024-04-23T18:17:09.537139+0000","last_fullsized":"2024-04-23T18:17:09.537139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:29.695226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546457+0000","last_change":"2024-04-23T18:15:05.688747+0000","last_active":"2024-04-23T18:17:09.546457+0000","last_peered":"2024-04-23T18:17:09.546457+0000","last_clean":"2024-04-23T18:17:09.546457+0000","last_became_active":"2024-04-23T18:15:05.688356+0000","last_became_peered":"2024-04-23T18:15:05.688356+0000","last_unstale":"2024-04-23T18:17:09.546457+0000","last_undegraded":"2024-04-23T18:17:09.546457+0000","last_fullsized":"2024-04-23T18:17:09.546457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:16.512413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543773+0000","last_change":"2024-04-23T18:15:06.449368+0000","last_active":"2024-04-23T18:17:09.543773+0000","last_peered":"2024-04-23T18:17:09.543773+0000","last_clean":"2024-04-23T18:17:09.543773+0000","last_became_active":"2024-04-23T18:15:06.449039+0000","last_became_peered":"2024-04-23T18:15:06.449039+0000","last_unstale":"2024-04-23T18:17:09.543773+0000","last_undegraded":"2024-04-23T18:17:09.543773+0000","last_fullsized":"2024-04-23T18:17:09.543773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:18:08.008590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589412+0000","last_change":"2024-04-23T18:15:05.690093+0000","last_active":"2024-04-23T18:17:09.589412+0000","last_peered":"2024-04-23T18:17:09.589412+0000","last_clean":"2024-04-23T18:17:09.589412+0000","last_became_active":"2024-04-23T18:15:05.689559+0000","last_became_peered":"2024-04-23T18:15:05.689559+0000","last_unstale":"2024-04-23T18:17:09.589412+0000","last_undegraded":"2024-04-23T18:17:09.589412+0000","last_fullsized":"2024-04-23T18:17:09.589412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:17.715026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537456+0000","last_change":"2024-04-23T18:15:06.446982+0000","last_active":"2024-04-23T18:17:09.537456+0000","last_peered":"2024-04-23T18:17:09.537456+0000","last_clean":"2024-04-23T18:17:09.537456+0000","last_became_active":"2024-04-23T18:15:06.446536+0000","last_became_peered":"2024-04-23T18:15:06.446536+0000","last_unstale":"2024-04-23T18:17:09.537456+0000","last_undegraded":"2024-04-23T18:17:09.537456+0000","last_fullsized":"2024-04-23T18:17:09.537456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:42.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543566+0000","last_change":"2024-04-23T18:15:05.690251+0000","last_active":"2024-04-23T18:17:09.543566+0000","last_peered":"2024-04-23T18:17:09.543566+0000","last_clean":"2024-04-23T18:17:09.543566+0000","last_became_active":"2024-04-23T18:15:05.689882+0000","last_became_peered":"2024-04-23T18:15:05.689882+0000","last_unstale":"2024-04-23T18:17:09.543566+0000","last_undegraded":"2024-04-23T18:17:09.543566+0000","last_fullsized":"2024-04-23T18:17:09.543566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:15.739071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440595+0000","last_change":"2024-04-23T18:15:06.448848+0000","last_active":"2024-04-23T18:17:09.440595+0000","last_peered":"2024-04-23T18:17:09.440595+0000","last_clean":"2024-04-23T18:17:09.440595+0000","last_became_active":"2024-04-23T18:15:06.448502+0000","last_became_peered":"2024-04-23T18:15:06.448502+0000","last_unstale":"2024-04-23T18:17:09.440595+0000","last_undegraded":"2024-04-23T18:17:09.440595+0000","last_fullsized":"2024-04-23T18:17:09.440595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:29.276261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":144,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.692105+0000","last_change":"2024-04-23T18:15:05.679690+0000","last_active":"2024-04-23T18:17:09.692105+0000","last_peered":"2024-04-23T18:17:09.692105+0000","last_clean":"2024-04-23T18:17:09.692105+0000","last_became_active":"2024-04-23T18:15:05.679294+0000","last_became_peered":"2024-04-23T18:15:05.679294+0000","last_unstale":"2024-04-23T18:17:09.692105+0000","last_undegraded":"2024-04-23T18:17:09.692105+0000","last_fullsized":"2024-04-23T18:17:09.692105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:26:56.035642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589181+0000","last_change":"2024-04-23T18:15:06.446906+0000","last_active":"2024-04-23T18:17:09.589181+0000","last_peered":"2024-04-23T18:17:09.589181+0000","last_clean":"2024-04-23T18:17:09.589181+0000","last_became_active":"2024-04-23T18:15:06.446447+0000","last_became_peered":"2024-04-23T18:15:06.446447+0000","last_unstale":"2024-04-23T18:17:09.589181+0000","last_undegraded":"2024-04-23T18:17:09.589181+0000","last_fullsized":"2024-04-23T18:17:09.589181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.966400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:05.692759+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:05.692391+0000","last_became_peered":"2024-04-23T18:15:05.692391+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:49.190953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.609071+0000","last_change":"2024-04-23T18:16:12.306548+0000","last_active":"2024-04-23T18:17:10.609071+0000","last_peered":"2024-04-23T18:17:10.609071+0000","last_clean":"2024-04-23T18:17:10.609071+0000","last_became_active":"2024-04-23T18:16:12.306227+0000","last_became_peered":"2024-04-23T18:16:12.306227+0000","last_unstale":"2024-04-23T18:17:10.609071+0000","last_undegraded":"2024-04-23T18:17:10.609071+0000","last_fullsized":"2024-04-23T18:17:10.609071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:49.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.544029+0000","last_change":"2024-04-23T18:15:05.683773+0000","last_active":"2024-04-23T18:17:09.544029+0000","last_peered":"2024-04-23T18:17:09.544029+0000","last_clean":"2024-04-23T18:17:09.544029+0000","last_became_active":"2024-04-23T18:15:05.683459+0000","last_became_peered":"2024-04-23T18:15:05.683459+0000","last_unstale":"2024-04-23T18:17:09.544029+0000","last_undegraded":"2024-04-23T18:17:09.544029+0000","last_fullsized":"2024-04-23T18:17:09.544029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:21.054673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589084+0000","last_change":"2024-04-23T18:16:12.304182+0000","last_active":"2024-04-23T18:17:09.589084+0000","last_peered":"2024-04-23T18:17:09.589084+0000","last_clean":"2024-04-23T18:17:09.589084+0000","last_became_active":"2024-04-23T18:16:12.303708+0000","last_became_peered":"2024-04-23T18:16:12.303708+0000","last_unstale":"2024-04-23T18:17:09.589084+0000","last_undegraded":"2024-04-23T18:17:09.589084+0000","last_fullsized":"2024-04-23T18:17:09.589084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:32.224284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.386516+0000","last_change":"2024-04-23T18:15:05.682632+0000","last_active":"2024-04-23T18:17:10.386516+0000","last_peered":"2024-04-23T18:17:10.386516+0000","last_clean":"2024-04-23T18:17:10.386516+0000","last_became_active":"2024-04-23T18:15:05.682108+0000","last_became_peered":"2024-04-23T18:15:05.682108+0000","last_unstale":"2024-04-23T18:17:10.386516+0000","last_undegraded":"2024-04-23T18:17:10.386516+0000","last_fullsized":"2024-04-23T18:17:10.386516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:32.650208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":191,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537357+0000","last_change":"2024-04-23T18:15:06.444481+0000","last_active":"2024-04-23T18:17:09.537357+0000","last_peered":"2024-04-23T18:17:09.537357+0000","last_clean":"2024-04-23T18:17:09.537357+0000","last_became_active":"2024-04-23T18:15:06.444129+0000","last_became_peered":"2024-04-23T18:15:06.444129+0000","last_unstale":"2024-04-23T18:17:09.537357+0000","last_undegraded":"2024-04-23T18:17:09.537357+0000","last_fullsized":"2024-04-23T18:17:09.537357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:18:48.186718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588956+0000","last_change":"2024-04-23T18:16:12.314402+0000","last_active":"2024-04-23T18:17:09.588956+0000","last_peered":"2024-04-23T18:17:09.588956+0000","last_clean":"2024-04-23T18:17:09.588956+0000","last_became_active":"2024-04-23T18:16:12.305588+0000","last_became_peered":"2024-04-23T18:16:12.305588+0000","last_unstale":"2024-04-23T18:17:09.588956+0000","last_undegraded":"2024-04-23T18:17:09.588956+0000","last_fullsized":"2024-04-23T18:17:09.588956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:15.413380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589319+0000","last_change":"2024-04-23T18:15:06.442340+0000","last_active":"2024-04-23T18:17:09.589319+0000","last_peered":"2024-04-23T18:17:09.589319+0000","last_clean":"2024-04-23T18:17:09.589319+0000","last_became_active":"2024-04-23T18:15:06.442009+0000","last_became_peered":"2024-04-23T18:15:06.442009+0000","last_unstale":"2024-04-23T18:17:09.589319+0000","last_undegraded":"2024-04-23T18:17:09.589319+0000","last_fullsized":"2024-04-23T18:17:09.589319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:51.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543781+0000","last_change":"2024-04-23T18:15:05.690502+0000","last_active":"2024-04-23T18:17:09.543781+0000","last_peered":"2024-04-23T18:17:09.543781+0000","last_clean":"2024-04-23T18:17:09.543781+0000","last_became_active":"2024-04-23T18:15:05.690028+0000","last_became_peered":"2024-04-23T18:15:05.690028+0000","last_unstale":"2024-04-23T18:17:09.543781+0000","last_undegraded":"2024-04-23T18:17:09.543781+0000","last_fullsized":"2024-04-23T18:17:09.543781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:08:05.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442441+0000","last_change":"2024-04-23T18:15:06.448535+0000","last_active":"2024-04-23T18:17:09.442441+0000","last_peered":"2024-04-23T18:17:09.442441+0000","last_clean":"2024-04-23T18:17:09.442441+0000","last_became_active":"2024-04-23T18:15:06.448176+0000","last_became_peered":"2024-04-23T18:15:06.448176+0000","last_unstale":"2024-04-23T18:17:09.442441+0000","last_undegraded":"2024-04-23T18:17:09.442441+0000","last_fullsized":"2024-04-23T18:17:09.442441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:18.230649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'8","reported_seq":154,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.858716+0000","last_change":"2024-04-23T18:15:05.688193+0000","last_active":"2024-04-23T18:17:09.858716+0000","last_peered":"2024-04-23T18:17:09.858716+0000","last_clean":"2024-04-23T18:17:09.858716+0000","last_became_active":"2024-04-23T18:15:05.687844+0000","last_became_peered":"2024-04-23T18:15:05.687844+0000","last_unstale":"2024-04-23T18:17:09.858716+0000","last_undegraded":"2024-04-23T18:17:09.858716+0000","last_fullsized":"2024-04-23T18:17:09.858716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:37:15.626756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.441991+0000","last_change":"2024-04-23T18:15:06.447184+0000","last_active":"2024-04-23T18:17:09.441991+0000","last_peered":"2024-04-23T18:17:09.441991+0000","last_clean":"2024-04-23T18:17:09.441991+0000","last_became_active":"2024-04-23T18:15:06.446864+0000","last_became_peered":"2024-04-23T18:15:06.446864+0000","last_unstale":"2024-04-23T18:17:09.441991+0000","last_undegraded":"2024-04-23T18:17:09.441991+0000","last_fullsized":"2024-04-23T18:17:09.441991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:21.581667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:05.686664+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:05.686285+0000","last_became_peered":"2024-04-23T18:15:05.686285+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:45.885047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543656+0000","last_change":"2024-04-23T18:15:06.452345+0000","last_active":"2024-04-23T18:17:09.543656+0000","last_peered":"2024-04-23T18:17:09.543656+0000","last_clean":"2024-04-23T18:17:09.543656+0000","last_became_active":"2024-04-23T18:15:06.451987+0000","last_became_peered":"2024-04-23T18:15:06.451987+0000","last_unstale":"2024-04-23T18:17:09.543656+0000","last_undegraded":"2024-04-23T18:17:09.543656+0000","last_fullsized":"2024-04-23T18:17:09.543656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:25:38.575497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.688476+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.688136+0000","last_became_peered":"2024-04-23T18:15:05.688136+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:51.837354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589193+0000","last_change":"2024-04-23T18:16:12.304948+0000","last_active":"2024-04-23T18:17:09.589193+0000","last_peered":"2024-04-23T18:17:09.589193+0000","last_clean":"2024-04-23T18:17:09.589193+0000","last_became_active":"2024-04-23T18:16:12.304541+0000","last_became_peered":"2024-04-23T18:16:12.304541+0000","last_unstale":"2024-04-23T18:17:09.589193+0000","last_undegraded":"2024-04-23T18:17:09.589193+0000","last_fullsized":"2024-04-23T18:17:09.589193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:47.148967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440865+0000","last_change":"2024-04-23T18:15:05.686001+0000","last_active":"2024-04-23T18:17:09.440865+0000","last_peered":"2024-04-23T18:17:09.440865+0000","last_clean":"2024-04-23T18:17:09.440865+0000","last_became_active":"2024-04-23T18:15:05.685635+0000","last_became_peered":"2024-04-23T18:15:05.685635+0000","last_unstale":"2024-04-23T18:17:09.440865+0000","last_undegraded":"2024-04-23T18:17:09.440865+0000","last_fullsized":"2024-04-23T18:17:09.440865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:28.143147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546656+0000","last_change":"2024-04-23T18:15:06.442592+0000","last_active":"2024-04-23T18:17:09.546656+0000","last_peered":"2024-04-23T18:17:09.546656+0000","last_clean":"2024-04-23T18:17:09.546656+0000","last_became_active":"2024-04-23T18:15:06.442255+0000","last_became_peered":"2024-04-23T18:15:06.442255+0000","last_unstale":"2024-04-23T18:17:09.546656+0000","last_undegraded":"2024-04-23T18:17:09.546656+0000","last_fullsized":"2024-04-23T18:17:09.546656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:57.334977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442224+0000","last_change":"2024-04-23T18:15:05.695666+0000","last_active":"2024-04-23T18:17:09.442224+0000","last_peered":"2024-04-23T18:17:09.442224+0000","last_clean":"2024-04-23T18:17:09.442224+0000","last_became_active":"2024-04-23T18:15:05.695241+0000","last_became_peered":"2024-04-23T18:15:05.695241+0000","last_unstale":"2024-04-23T18:17:09.442224+0000","last_undegraded":"2024-04-23T18:17:09.442224+0000","last_fullsized":"2024-04-23T18:17:09.442224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:41:08.355709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.546711+0000","last_change":"2024-04-23T18:16:12.304513+0000","last_active":"2024-04-23T18:17:09.546711+0000","last_peered":"2024-04-23T18:17:09.546711+0000","last_clean":"2024-04-23T18:17:09.546711+0000","last_became_active":"2024-04-23T18:16:12.304069+0000","last_became_peered":"2024-04-23T18:16:12.304069+0000","last_unstale":"2024-04-23T18:17:09.546711+0000","last_undegraded":"2024-04-23T18:17:09.546711+0000","last_fullsized":"2024-04-23T18:17:09.546711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:58.834388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543731+0000","last_change":"2024-04-23T18:15:06.453906+0000","last_active":"2024-04-23T18:17:09.543731+0000","last_peered":"2024-04-23T18:17:09.543731+0000","last_clean":"2024-04-23T18:17:09.543731+0000","last_became_active":"2024-04-23T18:15:06.453526+0000","last_became_peered":"2024-04-23T18:15:06.453526+0000","last_unstale":"2024-04-23T18:17:09.543731+0000","last_undegraded":"2024-04-23T18:17:09.543731+0000","last_fullsized":"2024-04-23T18:17:09.543731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:33:18.745421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.857714+0000","last_change":"2024-04-23T18:15:05.685954+0000","last_active":"2024-04-23T18:17:09.857714+0000","last_peered":"2024-04-23T18:17:09.857714+0000","last_clean":"2024-04-23T18:17:09.857714+0000","last_became_active":"2024-04-23T18:15:05.685464+0000","last_became_peered":"2024-04-23T18:15:05.685464+0000","last_unstale":"2024-04-23T18:17:09.857714+0000","last_undegraded":"2024-04-23T18:17:09.857714+0000","last_fullsized":"2024-04-23T18:17:09.857714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:48.921073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537763+0000","last_change":"2024-04-23T18:15:06.447651+0000","last_active":"2024-04-23T18:17:09.537763+0000","last_peered":"2024-04-23T18:17:09.537763+0000","last_clean":"2024-04-23T18:17:09.537763+0000","last_became_active":"2024-04-23T18:15:06.447238+0000","last_became_peered":"2024-04-23T18:15:06.447238+0000","last_unstale":"2024-04-23T18:17:09.537763+0000","last_undegraded":"2024-04-23T18:17:09.537763+0000","last_fullsized":"2024-04-23T18:17:09.537763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:51.018564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589454+0000","last_change":"2024-04-23T18:15:05.690004+0000","last_active":"2024-04-23T18:17:09.589454+0000","last_peered":"2024-04-23T18:17:09.589454+0000","last_clean":"2024-04-23T18:17:09.589454+0000","last_became_active":"2024-04-23T18:15:05.689688+0000","last_became_peered":"2024-04-23T18:15:05.689688+0000","last_unstale":"2024-04-23T18:17:09.589454+0000","last_undegraded":"2024-04-23T18:17:09.589454+0000","last_fullsized":"2024-04-23T18:17:09.589454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:42.929061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442087+0000","last_change":"2024-04-23T18:13:53.761742+0000","last_active":"2024-04-23T18:17:09.442087+0000","last_peered":"2024-04-23T18:17:09.442087+0000","last_clean":"2024-04-23T18:17:09.442087+0000","last_became_active":"2024-04-23T18:13:53.761276+0000","last_became_peered":"2024-04-23T18:13:53.761276+0000","last_unstale":"2024-04-23T18:17:09.442087+0000","last_undegraded":"2024-04-23T18:17:09.442087+0000","last_fullsized":"2024-04-23T18:17:09.442087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:10:35.673892+0000","last_clean_scrub_stamp":"2024-04-23T18:10:35.673892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:36.526242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589185+0000","last_change":"2024-04-23T18:15:06.437042+0000","last_active":"2024-04-23T18:17:09.589185+0000","last_peered":"2024-04-23T18:17:09.589185+0000","last_clean":"2024-04-23T18:17:09.589185+0000","last_became_active":"2024-04-23T18:15:06.436645+0000","last_became_peered":"2024-04-23T18:15:06.436645+0000","last_unstale":"2024-04-23T18:17:09.589185+0000","last_undegraded":"2024-04-23T18:17:09.589185+0000","last_fullsized":"2024-04-23T18:17:09.589185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:22.395713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442419+0000","last_change":"2024-04-23T18:15:05.689792+0000","last_active":"2024-04-23T18:17:09.442419+0000","last_peered":"2024-04-23T18:17:09.442419+0000","last_clean":"2024-04-23T18:17:09.442419+0000","last_became_active":"2024-04-23T18:15:05.689357+0000","last_became_peered":"2024-04-23T18:15:05.689357+0000","last_unstale":"2024-04-23T18:17:09.442419+0000","last_undegraded":"2024-04-23T18:17:09.442419+0000","last_fullsized":"2024-04-23T18:17:09.442419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.206288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537429+0000","last_change":"2024-04-23T18:15:05.696103+0000","last_active":"2024-04-23T18:17:09.537429+0000","last_peered":"2024-04-23T18:17:09.537429+0000","last_clean":"2024-04-23T18:17:09.537429+0000","last_became_active":"2024-04-23T18:15:05.695749+0000","last_became_peered":"2024-04-23T18:15:05.695749+0000","last_unstale":"2024-04-23T18:17:09.537429+0000","last_undegraded":"2024-04-23T18:17:09.537429+0000","last_fullsized":"2024-04-23T18:17:09.537429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:44:00.843200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589751+0000","last_change":"2024-04-23T18:15:06.440259+0000","last_active":"2024-04-23T18:17:09.589751+0000","last_peered":"2024-04-23T18:17:09.589751+0000","last_clean":"2024-04-23T18:17:09.589751+0000","last_became_active":"2024-04-23T18:15:06.439838+0000","last_became_peered":"2024-04-23T18:15:06.439838+0000","last_unstale":"2024-04-23T18:17:09.589751+0000","last_undegraded":"2024-04-23T18:17:09.589751+0000","last_fullsized":"2024-04-23T18:17:09.589751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:10.158817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'97","reported_seq":211,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:11.754963+0000","last_change":"2024-04-23T18:15:05.689339+0000","last_active":"2024-04-23T18:17:11.754963+0000","last_peered":"2024-04-23T18:17:11.754963+0000","last_clean":"2024-04-23T18:17:11.754963+0000","last_became_active":"2024-04-23T18:15:05.687800+0000","last_became_peered":"2024-04-23T18:15:05.687800+0000","last_unstale":"2024-04-23T18:17:11.754963+0000","last_undegraded":"2024-04-23T18:17:11.754963+0000","last_fullsized":"2024-04-23T18:17:11.754963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:38:33.646573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":177,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537883+0000","last_change":"2024-04-23T18:15:06.446320+0000","last_active":"2024-04-23T18:17:09.537883+0000","last_peered":"2024-04-23T18:17:09.537883+0000","last_clean":"2024-04-23T18:17:09.537883+0000","last_became_active":"2024-04-23T18:15:06.446002+0000","last_became_peered":"2024-04-23T18:15:06.446002+0000","last_unstale":"2024-04-23T18:17:09.537883+0000","last_undegraded":"2024-04-23T18:17:09.537883+0000","last_fullsized":"2024-04-23T18:17:09.537883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:06.897067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589819+0000","last_change":"2024-04-23T18:15:05.697891+0000","last_active":"2024-04-23T18:17:09.589819+0000","last_peered":"2024-04-23T18:17:09.589819+0000","last_clean":"2024-04-23T18:17:09.589819+0000","last_became_active":"2024-04-23T18:15:05.697536+0000","last_became_peered":"2024-04-23T18:15:05.697536+0000","last_unstale":"2024-04-23T18:17:09.589819+0000","last_undegraded":"2024-04-23T18:17:09.589819+0000","last_fullsized":"2024-04-23T18:17:09.589819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:33.383559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.609050+0000","last_change":"2024-04-23T18:15:06.451653+0000","last_active":"2024-04-23T18:17:10.609050+0000","last_peered":"2024-04-23T18:17:10.609050+0000","last_clean":"2024-04-23T18:17:10.609050+0000","last_became_active":"2024-04-23T18:15:06.450829+0000","last_became_peered":"2024-04-23T18:15:06.450829+0000","last_unstale":"2024-04-23T18:17:10.609050+0000","last_undegraded":"2024-04-23T18:17:10.609050+0000","last_fullsized":"2024-04-23T18:17:10.609050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:32:40.690082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588772+0000","last_change":"2024-04-23T18:15:05.685265+0000","last_active":"2024-04-23T18:17:09.588772+0000","last_peered":"2024-04-23T18:17:09.588772+0000","last_clean":"2024-04-23T18:17:09.588772+0000","last_became_active":"2024-04-23T18:15:05.684926+0000","last_became_peered":"2024-04-23T18:15:05.684926+0000","last_unstale":"2024-04-23T18:17:09.588772+0000","last_undegraded":"2024-04-23T18:17:09.588772+0000","last_fullsized":"2024-04-23T18:17:09.588772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:50.654367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537133+0000","last_change":"2024-04-23T18:15:06.453247+0000","last_active":"2024-04-23T18:17:09.537133+0000","last_peered":"2024-04-23T18:17:09.537133+0000","last_clean":"2024-04-23T18:17:09.537133+0000","last_became_active":"2024-04-23T18:15:06.452901+0000","last_became_peered":"2024-04-23T18:15:06.452901+0000","last_unstale":"2024-04-23T18:17:09.537133+0000","last_undegraded":"2024-04-23T18:17:09.537133+0000","last_fullsized":"2024-04-23T18:17:09.537133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:25.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537556+0000","last_change":"2024-04-23T18:15:05.690820+0000","last_active":"2024-04-23T18:17:09.537556+0000","last_peered":"2024-04-23T18:17:09.537556+0000","last_clean":"2024-04-23T18:17:09.537556+0000","last_became_active":"2024-04-23T18:15:05.689660+0000","last_became_peered":"2024-04-23T18:15:05.689660+0000","last_unstale":"2024-04-23T18:17:09.537556+0000","last_undegraded":"2024-04-23T18:17:09.537556+0000","last_fullsized":"2024-04-23T18:17:09.537556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.182296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589049+0000","last_change":"2024-04-23T18:15:06.452384+0000","last_active":"2024-04-23T18:17:09.589049+0000","last_peered":"2024-04-23T18:17:09.589049+0000","last_clean":"2024-04-23T18:17:09.589049+0000","last_became_active":"2024-04-23T18:15:06.451920+0000","last_became_peered":"2024-04-23T18:15:06.451920+0000","last_unstale":"2024-04-23T18:17:09.589049+0000","last_undegraded":"2024-04-23T18:17:09.589049+0000","last_fullsized":"2024-04-23T18:17:09.589049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:27:48.190620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442308+0000","last_change":"2024-04-23T18:15:05.694220+0000","last_active":"2024-04-23T18:17:09.442308+0000","last_peered":"2024-04-23T18:17:09.442308+0000","last_clean":"2024-04-23T18:17:09.442308+0000","last_became_active":"2024-04-23T18:15:05.693849+0000","last_became_peered":"2024-04-23T18:15:05.693849+0000","last_unstale":"2024-04-23T18:17:09.442308+0000","last_undegraded":"2024-04-23T18:17:09.442308+0000","last_fullsized":"2024-04-23T18:17:09.442308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:26.458041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.440818+0000","last_change":"2024-04-23T18:16:12.304138+0000","last_active":"2024-04-23T18:17:09.440818+0000","last_peered":"2024-04-23T18:17:09.440818+0000","last_clean":"2024-04-23T18:17:09.440818+0000","last_became_active":"2024-04-23T18:16:12.303760+0000","last_became_peered":"2024-04-23T18:16:12.303760+0000","last_unstale":"2024-04-23T18:17:09.440818+0000","last_undegraded":"2024-04-23T18:17:09.440818+0000","last_fullsized":"2024-04-23T18:17:09.440818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:10.049511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589223+0000","last_change":"2024-04-23T18:16:12.317810+0000","last_active":"2024-04-23T18:17:09.589223+0000","last_peered":"2024-04-23T18:17:09.589223+0000","last_clean":"2024-04-23T18:17:09.589223+0000","last_became_active":"2024-04-23T18:16:12.307064+0000","last_became_peered":"2024-04-23T18:16:12.307064+0000","last_unstale":"2024-04-23T18:17:09.589223+0000","last_undegraded":"2024-04-23T18:17:09.589223+0000","last_fullsized":"2024-04-23T18:17:09.589223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:06:41.038382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442209+0000","last_change":"2024-04-23T18:15:06.450772+0000","last_active":"2024-04-23T18:17:09.442209+0000","last_peered":"2024-04-23T18:17:09.442209+0000","last_clean":"2024-04-23T18:17:09.442209+0000","last_became_active":"2024-04-23T18:15:06.450466+0000","last_became_peered":"2024-04-23T18:15:06.450466+0000","last_unstale":"2024-04-23T18:17:09.442209+0000","last_undegraded":"2024-04-23T18:17:09.442209+0000","last_fullsized":"2024-04-23T18:17:09.442209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:10.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442303+0000","last_change":"2024-04-23T18:15:05.691098+0000","last_active":"2024-04-23T18:17:09.442303+0000","last_peered":"2024-04-23T18:17:09.442303+0000","last_clean":"2024-04-23T18:17:09.442303+0000","last_became_active":"2024-04-23T18:15:05.690692+0000","last_became_peered":"2024-04-23T18:15:05.690692+0000","last_unstale":"2024-04-23T18:17:09.442303+0000","last_undegraded":"2024-04-23T18:17:09.442303+0000","last_fullsized":"2024-04-23T18:17:09.442303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:40.267204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442516+0000","last_change":"2024-04-23T18:15:06.451943+0000","last_active":"2024-04-23T18:17:09.442516+0000","last_peered":"2024-04-23T18:17:09.442516+0000","last_clean":"2024-04-23T18:17:09.442516+0000","last_became_active":"2024-04-23T18:15:06.451547+0000","last_became_peered":"2024-04-23T18:15:06.451547+0000","last_unstale":"2024-04-23T18:17:09.442516+0000","last_undegraded":"2024-04-23T18:17:09.442516+0000","last_fullsized":"2024-04-23T18:17:09.442516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:13:00.985070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.650266+0000","last_change":"2024-04-23T18:15:05.682810+0000","last_active":"2024-04-23T18:17:09.650266+0000","last_peered":"2024-04-23T18:17:09.650266+0000","last_clean":"2024-04-23T18:17:09.650266+0000","last_became_active":"2024-04-23T18:15:05.682491+0000","last_became_peered":"2024-04-23T18:15:05.682491+0000","last_unstale":"2024-04-23T18:17:09.650266+0000","last_undegraded":"2024-04-23T18:17:09.650266+0000","last_fullsized":"2024-04-23T18:17:09.650266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:08.250621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442273+0000","last_change":"2024-04-23T18:15:06.446351+0000","last_active":"2024-04-23T18:17:09.442273+0000","last_peered":"2024-04-23T18:17:09.442273+0000","last_clean":"2024-04-23T18:17:09.442273+0000","last_became_active":"2024-04-23T18:15:06.445991+0000","last_became_peered":"2024-04-23T18:15:06.445991+0000","last_unstale":"2024-04-23T18:17:09.442273+0000","last_undegraded":"2024-04-23T18:17:09.442273+0000","last_fullsized":"2024-04-23T18:17:09.442273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:50.360655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589800+0000","last_change":"2024-04-23T18:15:05.684605+0000","last_active":"2024-04-23T18:17:09.589800+0000","last_peered":"2024-04-23T18:17:09.589800+0000","last_clean":"2024-04-23T18:17:09.589800+0000","last_became_active":"2024-04-23T18:15:05.684096+0000","last_became_peered":"2024-04-23T18:15:05.684096+0000","last_unstale":"2024-04-23T18:17:09.589800+0000","last_undegraded":"2024-04-23T18:17:09.589800+0000","last_fullsized":"2024-04-23T18:17:09.589800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:04.664538+0000","last_clean_scrub_stamp":"2024-04-23T18:15:04.664538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.794121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442682+0000","last_change":"2024-04-23T18:15:06.446257+0000","last_active":"2024-04-23T18:17:09.442682+0000","last_peered":"2024-04-23T18:17:09.442682+0000","last_clean":"2024-04-23T18:17:09.442682+0000","last_became_active":"2024-04-23T18:15:06.445690+0000","last_became_peered":"2024-04-23T18:15:06.445690+0000","last_unstale":"2024-04-23T18:17:09.442682+0000","last_undegraded":"2024-04-23T18:17:09.442682+0000","last_fullsized":"2024-04-23T18:17:09.442682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:43:01.995198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589247+0000","last_change":"2024-04-23T18:15:06.453451+0000","last_active":"2024-04-23T18:17:09.589247+0000","last_peered":"2024-04-23T18:17:09.589247+0000","last_clean":"2024-04-23T18:17:09.589247+0000","last_became_active":"2024-04-23T18:15:06.453028+0000","last_became_peered":"2024-04-23T18:15:06.453028+0000","last_unstale":"2024-04-23T18:17:09.589247+0000","last_undegraded":"2024-04-23T18:17:09.589247+0000","last_fullsized":"2024-04-23T18:17:09.589247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:29:51.277367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442275+0000","last_change":"2024-04-23T18:15:06.447024+0000","last_active":"2024-04-23T18:17:09.442275+0000","last_peered":"2024-04-23T18:17:09.442275+0000","last_clean":"2024-04-23T18:17:09.442275+0000","last_became_active":"2024-04-23T18:15:06.442002+0000","last_became_peered":"2024-04-23T18:15:06.442002+0000","last_unstale":"2024-04-23T18:17:09.442275+0000","last_undegraded":"2024-04-23T18:17:09.442275+0000","last_fullsized":"2024-04-23T18:17:09.442275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:52.659550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442192+0000","last_change":"2024-04-23T18:15:06.447401+0000","last_active":"2024-04-23T18:17:09.442192+0000","last_peered":"2024-04-23T18:17:09.442192+0000","last_clean":"2024-04-23T18:17:09.442192+0000","last_became_active":"2024-04-23T18:15:06.446553+0000","last_became_peered":"2024-04-23T18:15:06.446553+0000","last_unstale":"2024-04-23T18:17:09.442192+0000","last_undegraded":"2024-04-23T18:17:09.442192+0000","last_fullsized":"2024-04-23T18:17:09.442192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:56.818385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:10.608826+0000","last_change":"2024-04-23T18:15:06.450956+0000","last_active":"2024-04-23T18:17:10.608826+0000","last_peered":"2024-04-23T18:17:10.608826+0000","last_clean":"2024-04-23T18:17:10.608826+0000","last_became_active":"2024-04-23T18:15:06.450572+0000","last_became_peered":"2024-04-23T18:15:06.450572+0000","last_unstale":"2024-04-23T18:17:10.608826+0000","last_undegraded":"2024-04-23T18:17:10.608826+0000","last_fullsized":"2024-04-23T18:17:10.608826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:05.040651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589753+0000","last_change":"2024-04-23T18:15:06.448590+0000","last_active":"2024-04-23T18:17:09.589753+0000","last_peered":"2024-04-23T18:17:09.589753+0000","last_clean":"2024-04-23T18:17:09.589753+0000","last_became_active":"2024-04-23T18:15:06.448262+0000","last_became_peered":"2024-04-23T18:15:06.448262+0000","last_unstale":"2024-04-23T18:17:09.589753+0000","last_undegraded":"2024-04-23T18:17:09.589753+0000","last_fullsized":"2024-04-23T18:17:09.589753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:04:19.410256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589099+0000","last_change":"2024-04-23T18:15:06.446372+0000","last_active":"2024-04-23T18:17:09.589099+0000","last_peered":"2024-04-23T18:17:09.589099+0000","last_clean":"2024-04-23T18:17:09.589099+0000","last_became_active":"2024-04-23T18:15:06.445564+0000","last_became_peered":"2024-04-23T18:15:06.445564+0000","last_unstale":"2024-04-23T18:17:09.589099+0000","last_undegraded":"2024-04-23T18:17:09.589099+0000","last_fullsized":"2024-04-23T18:17:09.589099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:10:34.767484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.442534+0000","last_change":"2024-04-23T18:15:06.438304+0000","last_active":"2024-04-23T18:17:09.442534+0000","last_peered":"2024-04-23T18:17:09.442534+0000","last_clean":"2024-04-23T18:17:09.442534+0000","last_became_active":"2024-04-23T18:15:06.437938+0000","last_became_peered":"2024-04-23T18:15:06.437938+0000","last_unstale":"2024-04-23T18:17:09.442534+0000","last_undegraded":"2024-04-23T18:17:09.442534+0000","last_fullsized":"2024-04-23T18:17:09.442534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:23:05.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537737+0000","last_change":"2024-04-23T18:15:06.451920+0000","last_active":"2024-04-23T18:17:09.537737+0000","last_peered":"2024-04-23T18:17:09.537737+0000","last_clean":"2024-04-23T18:17:09.537737+0000","last_became_active":"2024-04-23T18:15:06.451429+0000","last_became_peered":"2024-04-23T18:15:06.451429+0000","last_unstale":"2024-04-23T18:17:09.537737+0000","last_undegraded":"2024-04-23T18:17:09.537737+0000","last_fullsized":"2024-04-23T18:17:09.537737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:58.443439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.543635+0000","last_change":"2024-04-23T18:15:06.447087+0000","last_active":"2024-04-23T18:17:09.543635+0000","last_peered":"2024-04-23T18:17:09.543635+0000","last_clean":"2024-04-23T18:17:09.543635+0000","last_became_active":"2024-04-23T18:15:06.446645+0000","last_became_peered":"2024-04-23T18:15:06.446645+0000","last_unstale":"2024-04-23T18:17:09.543635+0000","last_undegraded":"2024-04-23T18:17:09.543635+0000","last_fullsized":"2024-04-23T18:17:09.543635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:23.819988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.537500+0000","last_change":"2024-04-23T18:15:06.452958+0000","last_active":"2024-04-23T18:17:09.537500+0000","last_peered":"2024-04-23T18:17:09.537500+0000","last_clean":"2024-04-23T18:17:09.537500+0000","last_became_active":"2024-04-23T18:15:06.452593+0000","last_became_peered":"2024-04-23T18:15:06.452593+0000","last_unstale":"2024-04-23T18:17:09.537500+0000","last_undegraded":"2024-04-23T18:17:09.537500+0000","last_fullsized":"2024-04-23T18:17:09.537500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:41.614325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589280+0000","last_change":"2024-04-23T18:15:06.441229+0000","last_active":"2024-04-23T18:17:09.589280+0000","last_peered":"2024-04-23T18:17:09.589280+0000","last_clean":"2024-04-23T18:17:09.589280+0000","last_became_active":"2024-04-23T18:15:06.440898+0000","last_became_peered":"2024-04-23T18:15:06.440898+0000","last_unstale":"2024-04-23T18:17:09.589280+0000","last_undegraded":"2024-04-23T18:17:09.589280+0000","last_fullsized":"2024-04-23T18:17:09.589280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.335101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.589566+0000","last_change":"2024-04-23T18:15:06.443075+0000","last_active":"2024-04-23T18:17:09.589566+0000","last_peered":"2024-04-23T18:17:09.589566+0000","last_clean":"2024-04-23T18:17:09.589566+0000","last_became_active":"2024-04-23T18:15:06.442741+0000","last_became_peered":"2024-04-23T18:15:06.442741+0000","last_unstale":"2024-04-23T18:17:09.589566+0000","last_undegraded":"2024-04-23T18:17:09.589566+0000","last_fullsized":"2024-04-23T18:17:09.589566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:49.280863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:09.588942+0000","last_change":"2024-04-23T18:15:06.447438+0000","last_active":"2024-04-23T18:17:09.588942+0000","last_peered":"2024-04-23T18:17:09.588942+0000","last_clean":"2024-04-23T18:17:09.588942+0000","last_became_active":"2024-04-23T18:15:06.447062+0000","last_became_peered":"2024-04-23T18:15:06.447062+0000","last_unstale":"2024-04-23T18:17:09.588942+0000","last_undegraded":"2024-04-23T18:17:09.588942+0000","last_fullsized":"2024-04-23T18:17:09.588942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:15:05.666879+0000","last_clean_scrub_stamp":"2024-04-23T18:15:05.666879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:46.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598588,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1066,"num_read_kb":736,"num_write":385,"num_write_kb":1885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":656,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1795764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":469660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1114623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776177,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36684,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708468,"statfs":{"total":95995035648,"available":95957471232,"internally_reserved":0,"allocated":1380352,"data_stored":863078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704108,"statfs":{"total":95995035648,"available":95953006592,"internally_reserved":0,"allocated":1581056,"data_stored":1063725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":536802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":995328,"data_stored":470522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":537496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":994378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40464,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704688,"statfs":{"total":95995035648,"available":95953600512,"internally_reserved":0,"allocated":987136,"data_stored":472714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712108,"statfs":{"total":95995035648,"available":95961198592,"internally_reserved":0,"allocated":1728512,"data_stored":1192605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:17:15.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 -- 172.21.15.26:0/1651511826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc250102bf0 msgr2=0x7fc250106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1651511826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc250102bf0 0x7fc250106ba0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc238009900 tx=0x7fc23802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 -- 172.21.15.26:0/1651511826 shutdown_connections 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1651511826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 0x7fc25010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1651511826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc250102bf0 0x7fc250106ba0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1651511826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc250102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 -- 172.21.15.26:0/1651511826 >> 172.21.15.26:0/1651511826 conn(0x7fc2500790e0 msgr2=0x7fc2500794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 -- 172.21.15.26:0/1651511826 shutdown_connections 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.763+0000 7fc2559e1640 1 -- 172.21.15.26:0/1651511826 wait complete. 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 Processor -- start 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 -- start start 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc2501986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 0x7fc250198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc2501121f0 con 0x7fc25019cf90 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc250112070 con 0x7fc250107280 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc2559e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc250112370 con 0x7fc250102240 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc24f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc24f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48990/0 (socket says 172.21.15.26:48990) 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc24f7fe640 1 -- 172.21.15.26:0/1798698879 learned_addr learned my addr 172.21.15.26:0/1798698879 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:15.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc247fff640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 0x7fc250198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.764+0000 7fc24effd640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc2501986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 -- 172.21.15.26:0/1798698879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 msgr2=0x7fc2501986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc2501986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 -- 172.21.15.26:0/1798698879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 msgr2=0x7fc250198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 0x7fc250198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 -- 172.21.15.26:0/1798698879 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25019da80 con 0x7fc25019cf90 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24effd640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc2501986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc24f7fe640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc24800eaa0 tx=0x7fc24800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24800cdb0 con 0x7fc25019cf90 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc248005020 con 0x7fc25019cf90 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480116d0 con 0x7fc25019cf90 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25019dd70 con 0x7fc25019cf90 2024-04-23T18:17:15.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.765+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2501a54e0 con 0x7fc25019cf90 2024-04-23T18:17:15.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.766+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2500691c0 con 0x7fc25019cf90 2024-04-23T18:17:15.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.767+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fc2480051d0 con 0x7fc25019cf90 2024-04-23T18:17:15.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.768+0000 7fc2549df640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 0x7fc23c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:15.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.768+0000 7fc24effd640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 0x7fc23c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.768+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc2480d0f60 con 0x7fc25019cf90 2024-04-23T18:17:15.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.768+0000 7fc24effd640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 0x7fc23c0a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc24000baa0 tx=0x7fc240005d90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:15.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.771+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc248016030 con 0x7fc25019cf90 2024-04-23T18:17:15.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:15 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3881738296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:15.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:15 smithi110 ceph-mon[32585]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:15.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:15 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3881738296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:15.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:15 smithi026 ceph-mon[29364]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:15 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3881738296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:15.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:15 smithi131 ceph-mon[33041]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.997+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc250102bf0 con 0x7fc25019cf90 2024-04-23T18:17:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:15.998+0000 7fc2549df640 1 -- 172.21.15.26:0/1798698879 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11884 (secure 0 0 0) 0x7fc24809d460 con 0x7fc25019cf90 2024-04-23T18:17:15.993 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:15.993 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:08.076905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:17:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 22 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 msgr2=0x7fc23c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 0x7fc23c0a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc24000baa0 tx=0x7fc240005d90 comp rx=0 tx=0).stop 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 msgr2=0x7fc25019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc24800eaa0 tx=0x7fc24800ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 shutdown_connections 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc23c0a3db0 0x7fc23c0a6270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250107280 0x7fc250198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc25019cf90 0x7fc25019d440 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 --2- 172.21.15.26:0/1798698879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc250102240 0x7fc2501986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 >> 172.21.15.26:0/1798698879 conn(0x7fc2500790e0 msgr2=0x7fc2501000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 shutdown_connections 2024-04-23T18:17:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.000+0000 7fc2559e1640 1 -- 172.21.15.26:0/1798698879 wait complete. 2024-04-23T18:17:16.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 -- 172.21.15.26:0/3763078728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f410c520 msgr2=0x7fb4f410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 --2- 172.21.15.26:0/3763078728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f410c520 0x7fb4f410e9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009ca0 tx=0x7fb4ec02a9e0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 -- 172.21.15.26:0/3763078728 shutdown_connections 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 --2- 172.21.15.26:0/3763078728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 0x7fb4f410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 --2- 172.21.15.26:0/3763078728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f410c520 0x7fb4f410e9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 --2- 172.21.15.26:0/3763078728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4f41031e0 0x7fb4f41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 -- 172.21.15.26:0/3763078728 >> 172.21.15.26:0/3763078728 conn(0x7fb4f4077f50 msgr2=0x7fb4f4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 -- 172.21.15.26:0/3763078728 shutdown_connections 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.114+0000 7fb4fb168640 1 -- 172.21.15.26:0/3763078728 wait complete. 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 Processor -- start 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 -- start start 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 0x7fb4f4198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4f410c520 0x7fb4f419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4112210 con 0x7fb4f41031e0 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4112090 con 0x7fb4f4103b00 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4fb168640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f4112390 con 0x7fb4f410c520 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49022/0 (socket says 172.21.15.26:49022) 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 -- 172.21.15.26:0/964005068 learned_addr learned my addr 172.21.15.26:0/964005068 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:16.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f3fff640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 0x7fb4f4198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 -- 172.21.15.26:0/964005068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4f410c520 msgr2=0x7fb4f419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4f410c520 0x7fb4f419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 -- 172.21.15.26:0/964005068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 msgr2=0x7fb4f4198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 0x7fb4f4198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 -- 172.21.15.26:0/964005068 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f419d690 con 0x7fb4f41031e0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.115+0000 7fb4f8f04640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800b4a0 tx=0x7fb4e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.116+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8014070 con 0x7fb4f41031e0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.116+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f419d980 con 0x7fb4f41031e0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.116+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4e8010ac0 con 0x7fb4f41031e0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.116+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8017c30 con 0x7fb4f41031e0 2024-04-23T18:17:16.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.116+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41a5210 con 0x7fb4f41031e0 2024-04-23T18:17:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.117+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f4068d70 con 0x7fb4f41031e0 2024-04-23T18:17:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.118+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb4e8017070 con 0x7fb4f41031e0 2024-04-23T18:17:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.119+0000 7fb4f1ffb640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 0x7fb4e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.119+0000 7fb4f3fff640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 0x7fb4e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.119+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb4e80cfff0 con 0x7fb4f41031e0 2024-04-23T18:17:16.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.119+0000 7fb4f3fff640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 0x7fb4e40a6270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0005e10 tx=0x7fb4e0005da0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.123+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4e809c660 con 0x7fb4f41031e0 2024-04-23T18:17:16.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.311+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4f4107110 con 0x7fb4f41031e0 2024-04-23T18:17:16.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.313+0000 7fb4f1ffb640 1 -- 172.21.15.26:0/964005068 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23037 (secure 0 0 0) 0x7fb4e809bf60 con 0x7fb4f41031e0 2024-04-23T18:17:16.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:16.308 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:17:09.435910+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:16.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 msgr2=0x7fb4e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 0x7fb4e40a6270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0005e10 tx=0x7fb4e0005da0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 msgr2=0x7fb4f41986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800b4a0 tx=0x7fb4e800b960 comp rx=0 tx=0).stop 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 shutdown_connections 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb4e40a3db0 0x7fb4e40a6270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4f4103b00 0x7fb4f4198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4f410c520 0x7fb4f419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 --2- 172.21.15.26:0/964005068 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb4f41031e0 0x7fb4f41986f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 >> 172.21.15.26:0/964005068 conn(0x7fb4f4077f50 msgr2=0x7fb4f40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 shutdown_connections 2024-04-23T18:17:16.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.315+0000 7fb4fb168640 1 -- 172.21.15.26:0/964005068 wait complete. 2024-04-23T18:17:16.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:17:16.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- 172.21.15.26:0/2378196263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 msgr2=0x7f38bc1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- 172.21.15.26:0/2378196263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc1035b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009900 tx=0x7f38ac02a510 comp rx=0 tx=0).stop 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- 172.21.15.26:0/2378196263 shutdown_connections 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- 172.21.15.26:0/2378196263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc103af0 0x7f38bc10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- 172.21.15.26:0/2378196263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc10c510 0x7f38bc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- 172.21.15.26:0/2378196263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc1035b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- 172.21.15.26:0/2378196263 >> 172.21.15.26:0/2378196263 conn(0x7f38bc077f50 msgr2=0x7f38bc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- 172.21.15.26:0/2378196263 shutdown_connections 2024-04-23T18:17:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- 172.21.15.26:0/2378196263 wait complete. 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 Processor -- start 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 -- start start 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 0x7f38bc101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.433+0000 7f38c324e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 0x7f38bc1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c324e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1121a0 con 0x7f38bc1031d0 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c324e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc112020 con 0x7f38bc10c510 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c324e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc112320 con 0x7f38bc103af0 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49040/0 (socket says 172.21.15.26:49040) 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 -- 172.21.15.26:0/4006884970 learned_addr learned my addr 172.21.15.26:0/4006884970 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38bbfff640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 0x7f38bc101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c17eb640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 0x7f38bc1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 -- 172.21.15.26:0/4006884970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 msgr2=0x7f38bc101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 0x7f38bc101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 -- 172.21.15.26:0/4006884970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 msgr2=0x7f38bc1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 0x7f38bc1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 -- 172.21.15.26:0/4006884970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38bc1a6c40 con 0x7f38bc1031d0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38bbfff640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 0x7f38bc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c17eb640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 0x7f38bc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c0fea640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f38ac005940 tx=0x7f38ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac03f030 con 0x7f38bc1031d0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38ac003aa0 con 0x7f38bc1031d0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38bc1a6ed0 con 0x7f38bc1031d0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac035070 con 0x7f38bc1031d0 2024-04-23T18:17:16.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.434+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38bc1a7330 con 0x7f38bc1031d0 2024-04-23T18:17:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.435+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38bc068d70 con 0x7f38bc1031d0 2024-04-23T18:17:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.437+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f38ac042030 con 0x7f38bc1031d0 2024-04-23T18:17:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.437+0000 7f38b9ffb640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 0x7f38a80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.437+0000 7f38bbfff640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 0x7f38a80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.437+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f38ac0f1250 con 0x7f38bc1031d0 2024-04-23T18:17:16.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.438+0000 7f38bbfff640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 0x7f38a80a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38bc1028b0 tx=0x7f38a4005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.441+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38ac0bd7b0 con 0x7f38bc1031d0 2024-04-23T18:17:16.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.629+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38bc1076c0 con 0x7f38bc1031d0 2024-04-23T18:17:16.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.629+0000 7f38b9ffb640 1 -- 172.21.15.26:0/4006884970 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f38ac0bd0b0 con 0x7f38bc1031d0 2024-04-23T18:17:16.624 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:17:16.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.631+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 msgr2=0x7f38a80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.631+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 0x7f38a80a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38bc1028b0 tx=0x7f38a4005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 msgr2=0x7f38bc100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f38ac005940 tx=0x7f38ac004270 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 shutdown_connections 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f38a80a3db0 0x7f38a80a6270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38bc10c510 0x7f38bc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38bc103af0 0x7f38bc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 --2- 172.21.15.26:0/4006884970 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f38bc1031d0 0x7f38bc100e20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 >> 172.21.15.26:0/4006884970 conn(0x7f38bc077f50 msgr2=0x7f38bc0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 shutdown_connections 2024-04-23T18:17:16.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.632+0000 7f38c324e640 1 -- 172.21.15.26:0/4006884970 wait complete. 2024-04-23T18:17:16.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:17:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- 172.21.15.26:0/2308754756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 msgr2=0x7fad2410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 --2- 172.21.15.26:0/2308754756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad2410bfc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fad18009900 tx=0x7fad1802a510 comp rx=0 tx=0).stop 2024-04-23T18:17:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- 172.21.15.26:0/2308754756 shutdown_connections 2024-04-23T18:17:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 --2- 172.21.15.26:0/2308754756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 --2- 172.21.15.26:0/2308754756 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad2410bfc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 --2- 172.21.15.26:0/2308754756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 0x7fad241035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- 172.21.15.26:0/2308754756 >> 172.21.15.26:0/2308754756 conn(0x7fad24078000 msgr2=0x7fad24078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- 172.21.15.26:0/2308754756 shutdown_connections 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- 172.21.15.26:0/2308754756 wait complete. 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 Processor -- start 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.750+0000 7fad297e1640 1 -- start start 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 0x7fad24198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112170 con 0x7fad24103ae0 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111ff0 con 0x7fad2410c500 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fad241122f0 con 0x7fad241031c0 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49068/0 (socket says 172.21.15.26:49068) 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 0x7fad24198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 -- 172.21.15.26:0/3646901438 learned_addr learned my addr 172.21.15.26:0/3646901438 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad237fe640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 -- 172.21.15.26:0/3646901438 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 msgr2=0x7fad24198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 0x7fad24198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 -- 172.21.15.26:0/3646901438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 msgr2=0x7fad2419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 -- 172.21.15.26:0/3646901438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2419d630 con 0x7fad24103ae0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad237fe640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad227fc640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad18005940 tx=0x7fad18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1803f030 con 0x7fad24103ae0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad18003aa0 con 0x7fad24103ae0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.751+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2419d8c0 con 0x7fad24103ae0 2024-04-23T18:17:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.752+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18035070 con 0x7fad24103ae0 2024-04-23T18:17:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.752+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241a51b0 con 0x7fad24103ae0 2024-04-23T18:17:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.752+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad24068d70 con 0x7fad24103ae0 2024-04-23T18:17:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.754+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fad18042030 con 0x7fad24103ae0 2024-04-23T18:17:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.754+0000 7fad0bfff640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 0x7fad0c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.754+0000 7fad22ffd640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 0x7fad0c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.754+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fad180f0fc0 con 0x7fad24103ae0 2024-04-23T18:17:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.755+0000 7fad22ffd640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 0x7fad0c0a62f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad1400abe0 tx=0x7fad1400a2d0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.759+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad180bd520 con 0x7fad24103ae0 2024-04-23T18:17:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.944+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad241070f0 con 0x7fad24103ae0 2024-04-23T18:17:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.945+0000 7fad0bfff640 1 -- 172.21.15.26:0/3646901438 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fad180bce20 con 0x7fad24103ae0 2024-04-23T18:17:16.939 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:17:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 msgr2=0x7fad0c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 0x7fad0c0a62f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad1400abe0 tx=0x7fad1400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 msgr2=0x7fad24198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad18005940 tx=0x7fad18004270 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 shutdown_connections 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad0c0a3e30 0x7fad0c0a62f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2410c500 0x7fad2419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad24103ae0 0x7fad24198bd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 --2- 172.21.15.26:0/3646901438 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad241031c0 0x7fad24198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 >> 172.21.15.26:0/3646901438 conn(0x7fad24078000 msgr2=0x7fad240fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 shutdown_connections 2024-04-23T18:17:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:16.947+0000 7fad297e1640 1 -- 172.21.15.26:0/3646901438 wait complete. 2024-04-23T18:17:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:16 smithi110 ceph-mon[32585]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:16 smithi110 ceph-mon[32585]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:17:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:16 smithi110 ceph-mon[32585]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1798698879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:16.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/964005068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:16.951 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:16 smithi026 ceph-mon[29364]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.951 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:16 smithi026 ceph-mon[29364]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:17:16.952 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:16 smithi026 ceph-mon[29364]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.952 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1798698879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:16.952 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/964005068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:16.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:17:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:16 smithi131 ceph-mon[33041]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:16 smithi131 ceph-mon[33041]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:17:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:16 smithi131 ceph-mon[33041]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1798698879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:16.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/964005068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:17.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- 172.21.15.26:0/2348091740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 msgr2=0x7fa52410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 --2- 172.21.15.26:0/2348091740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa52410ee30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009ca0 tx=0x7fa51c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- 172.21.15.26:0/2348091740 shutdown_connections 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 --2- 172.21.15.26:0/2348091740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa524102b40 0x7fa524106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 --2- 172.21.15.26:0/2348091740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa52410ee30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 --2- 172.21.15.26:0/2348091740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 0x7fa524102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- 172.21.15.26:0/2348091740 >> 172.21.15.26:0/2348091740 conn(0x7fa5240fd990 msgr2=0x7fa5240ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- 172.21.15.26:0/2348091740 shutdown_connections 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- 172.21.15.26:0/2348091740 wait complete. 2024-04-23T18:17:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 Processor -- start 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.064+0000 7fa529c43640 1 -- start start 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 0x7fa5241a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 0x7fa52419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa524075610 con 0x7fa524107260 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa524075490 con 0x7fa52419a750 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa524075790 con 0x7fa524102190 2024-04-23T18:17:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49094/0 (socket says 172.21.15.26:49094) 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 -- 172.21.15.26:0/885013151 learned_addr learned my addr 172.21.15.26:0/885013151 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa5237fe640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 0x7fa5241a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa523fff640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 0x7fa52419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 -- 172.21.15.26:0/885013151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 msgr2=0x7fa5241a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 0x7fa5241a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 -- 172.21.15.26:0/885013151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 msgr2=0x7fa52419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 0x7fa52419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 -- 172.21.15.26:0/885013151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52419b350 con 0x7fa524107260 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa523fff640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 0x7fa52419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa522ffd640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e9c0 tx=0x7fa51800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51800cd80 con 0x7fa524107260 2024-04-23T18:17:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5180054d0 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518010430 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.065+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5241a7430 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.066+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5241a7830 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.067+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa524103660 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.068+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fa518004fd0 con 0x7fa524107260 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.068+0000 7fa520ff9640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 0x7fa50c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.068+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa5180d0e20 con 0x7fa524107260 2024-04-23T18:17:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.069+0000 7fa5237fe640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 0x7fa50c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.069+0000 7fa5237fe640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 0x7fa50c0a6270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa514005fd0 tx=0x7fa51400b590 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.073+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa51809d380 con 0x7fa524107260 2024-04-23T18:17:17.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.259+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa5240691c0 con 0x7fa524107260 2024-04-23T18:17:17.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.260+0000 7fa520ff9640 1 -- 172.21.15.26:0/885013151 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa51809cc80 con 0x7fa524107260 2024-04-23T18:17:17.254 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:17:17.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 msgr2=0x7fa50c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 0x7fa50c0a6270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa514005fd0 tx=0x7fa51400b590 comp rx=0 tx=0).stop 2024-04-23T18:17:17.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 msgr2=0x7fa5241a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e9c0 tx=0x7fa51800ee80 comp rx=0 tx=0).stop 2024-04-23T18:17:17.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 shutdown_connections 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa50c0a3db0 0x7fa50c0a6270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52419a750 0x7fa52419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa524107260 0x7fa5241a0bd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 --2- 172.21.15.26:0/885013151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa524102190 0x7fa5241a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 >> 172.21.15.26:0/885013151 conn(0x7fa5240fd990 msgr2=0x7fa5240fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 shutdown_connections 2024-04-23T18:17:17.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.262+0000 7fa529c43640 1 -- 172.21.15.26:0/885013151 wait complete. 2024-04-23T18:17:17.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 -- 172.21.15.26:0/2403531212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 msgr2=0x7fb95810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/2403531212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95810e9e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb950009ca0 tx=0x7fb95002a950 comp rx=0 tx=0).stop 2024-04-23T18:17:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 -- 172.21.15.26:0/2403531212 shutdown_connections 2024-04-23T18:17:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/2403531212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958103b50 0x7fb95810c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/2403531212 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95810e9e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/2403531212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 0x7fb958103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 -- 172.21.15.26:0/2403531212 >> 172.21.15.26:0/2403531212 conn(0x7fb958077fb0 msgr2=0x7fb9580783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 -- 172.21.15.26:0/2403531212 shutdown_connections 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.375+0000 7fb95d3bf640 1 -- 172.21.15.26:0/2403531212 wait complete. 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 Processor -- start 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 -- start start 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 0x7fb958198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958103b50 0x7fb958198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb9581121b0 con 0x7fb95810c570 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb958112030 con 0x7fb958103b50 2024-04-23T18:17:17.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb95d3bf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb958112330 con 0x7fb958103230 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49102/0 (socket says 172.21.15.26:49102) 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 -- 172.21.15.26:0/581118109 learned_addr learned my addr 172.21.15.26:0/581118109 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb956ffd640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 0x7fb958198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 -- 172.21.15.26:0/581118109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 msgr2=0x7fb958198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 0x7fb958198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 -- 172.21.15.26:0/581118109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958103b50 msgr2=0x7fb958198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958103b50 0x7fb958198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 -- 172.21.15.26:0/581118109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95819d640 con 0x7fb95810c570 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.376+0000 7fb9577fe640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb950004af0 tx=0x7fb95002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.377+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb950034040 con 0x7fb95810c570 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.377+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb95002fe80 con 0x7fb95810c570 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.377+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9500046a0 con 0x7fb95810c570 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.377+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95819d8d0 con 0x7fb95810c570 2024-04-23T18:17:17.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.377+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9581a51c0 con 0x7fb95810c570 2024-04-23T18:17:17.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.378+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9580691c0 con 0x7fb95810c570 2024-04-23T18:17:17.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.379+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb950037020 con 0x7fb95810c570 2024-04-23T18:17:17.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.380+0000 7fb94ffff640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 0x7fb9440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.380+0000 7fb956ffd640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 0x7fb9440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.380+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb9500f1700 con 0x7fb95810c570 2024-04-23T18:17:17.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.380+0000 7fb956ffd640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 0x7fb9440a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb940005e10 tx=0x7fb940005da0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.384+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9500bdda0 con 0x7fb95810c570 2024-04-23T18:17:17.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.611+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9581999a0 con 0x7fb95810c570 2024-04-23T18:17:17.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.612+0000 7fb94ffff640 1 -- 172.21.15.26:0/581118109 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11884 (secure 0 0 0) 0x7fb9581999a0 con 0x7fb95810c570 2024-04-23T18:17:17.607 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:17.607 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:08.076905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:17:17.607 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 22 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 msgr2=0x7fb9440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 0x7fb9440a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb940005e10 tx=0x7fb940005da0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 msgr2=0x7fb95819cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb950004af0 tx=0x7fb95002f5d0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 shutdown_connections 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb9440a3e30 0x7fb9440a62f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958103b50 0x7fb958198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb95810c570 0x7fb95819cf10 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 --2- 172.21.15.26:0/581118109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb958103230 0x7fb958198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 >> 172.21.15.26:0/581118109 conn(0x7fb958077fb0 msgr2=0x7fb9580fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 shutdown_connections 2024-04-23T18:17:17.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.614+0000 7fb95d3bf640 1 -- 172.21.15.26:0/581118109 wait complete. 2024-04-23T18:17:17.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- 172.21.15.26:0/3679208665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca8103b00 msgr2=0x7efca810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 --2- 172.21.15.26:0/3679208665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca8103b00 0x7efca810bfe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7efc98009900 tx=0x7efc9802a510 comp rx=0 tx=0).stop 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- 172.21.15.26:0/3679208665 shutdown_connections 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 --2- 172.21.15.26:0/3679208665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca810c520 0x7efca810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 --2- 172.21.15.26:0/3679208665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca8103b00 0x7efca810bfe0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 --2- 172.21.15.26:0/3679208665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- 172.21.15.26:0/3679208665 >> 172.21.15.26:0/3679208665 conn(0x7efca8077f50 msgr2=0x7efca8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- 172.21.15.26:0/3679208665 shutdown_connections 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- 172.21.15.26:0/3679208665 wait complete. 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 Processor -- start 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.728+0000 7efcaf26b640 1 -- start start 2024-04-23T18:17:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca8103b00 0x7efca8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7efca81121b0 con 0x7efca810c520 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efca8112030 con 0x7efca81031e0 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7efca8112330 con 0x7efca8103b00 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49128/0 (socket says 172.21.15.26:49128) 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 -- 172.21.15.26:0/1016333371 learned_addr learned my addr 172.21.15.26:0/1016333371 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcac806640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca8103b00 0x7efca8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad007640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 -- 172.21.15.26:0/1016333371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca8103b00 msgr2=0x7efca8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca8103b00 0x7efca8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 -- 172.21.15.26:0/1016333371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 msgr2=0x7efca8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 -- 172.21.15.26:0/1016333371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca819d720 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad007640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcad808640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efca000eb10 tx=0x7efca000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0004b20 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca0004cc0 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0010670 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.729+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca819da10 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.730+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca81a52a0 con 0x7efca810c520 2024-04-23T18:17:17.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.730+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca8068d70 con 0x7efca810c520 2024-04-23T18:17:17.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.731+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7efca0005020 con 0x7efca810c520 2024-04-23T18:17:17.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.732+0000 7efc9e7fc640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 0x7efc940a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.732+0000 7efcad007640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 0x7efc940a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.732+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7efca00d0e90 con 0x7efca810c520 2024-04-23T18:17:17.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.733+0000 7efcad007640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 0x7efc940a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efca4009c20 tx=0x7efca4005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.736+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efca009d390 con 0x7efca810c520 2024-04-23T18:17:17.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.925+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efca8107110 con 0x7efca810c520 2024-04-23T18:17:17.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.926+0000 7efc9e7fc640 1 -- 172.21.15.26:0/1016333371 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23037 (secure 0 0 0) 0x7efca009cc90 con 0x7efca810c520 2024-04-23T18:17:17.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:17.921 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":89,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:17:09.435910+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.928+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 msgr2=0x7efc940a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.928+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 0x7efc940a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efca4009c20 tx=0x7efca4005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:17.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.928+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 msgr2=0x7efca819cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.928+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efca000eb10 tx=0x7efca000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 shutdown_connections 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efc940a3e30 0x7efc940a62f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efca810c520 0x7efca819cff0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efca8103b00 0x7efca8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 --2- 172.21.15.26:0/1016333371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efca81031e0 0x7efca8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 >> 172.21.15.26:0/1016333371 conn(0x7efca8077f50 msgr2=0x7efca80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 shutdown_connections 2024-04-23T18:17:17.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:17.929+0000 7efcaf26b640 1 -- 172.21.15.26:0/1016333371 wait complete. 2024-04-23T18:17:17.933 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4006884970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.933 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3646901438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.933 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:17 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/885013151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.935 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 117392 2024-04-23T18:17:17.935 DEBUG:tasks.quiescer:Yielding 2024-04-23T18:17:17.935 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T18:17:17.950 INFO:tasks.workunit:Pulling workunits from ref 57e0caf76fdb7ab8b1358588f292d08519163844 2024-04-23T18:17:17.951 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T18:17:17.951 INFO:tasks.workunit:timeout=3h 2024-04-23T18:17:17.951 INFO:tasks.workunit:cleanup=True 2024-04-23T18:17:17.951 DEBUG:teuthology.orchestra.run.smithi026:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:17.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:17.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4006884970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3646901438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:17 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/885013151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-23 18:15:13.092515527 +0000 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-23 18:15:13.092515527 +0000 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-23 18:17:07.379052644 +0000 2024-04-23T18:17:17.968 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-23 18:15:13.092515527 +0000 2024-04-23T18:17:17.969 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-23T18:17:17.969 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-23T18:17:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4006884970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3646901438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:17.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:17 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/885013151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:18.038 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 57e0caf76fdb7ab8b1358588f292d08519163844 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 -- 172.21.15.26:0/3710972120 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad80101ac0 msgr2=0x7fad80105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 --2- 172.21.15.26:0/3710972120 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad80101ac0 0x7fad80105ab0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fad70009900 tx=0x7fad7002a410 comp rx=0 tx=0).stop 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 -- 172.21.15.26:0/3710972120 shutdown_connections 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 --2- 172.21.15.26:0/3710972120 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80106190 0x7fad8010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 --2- 172.21.15.26:0/3710972120 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad80101ac0 0x7fad80105ab0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 --2- 172.21.15.26:0/3710972120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80101110 0x7fad801014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 -- 172.21.15.26:0/3710972120 >> 172.21.15.26:0/3710972120 conn(0x7fad80077fb0 msgr2=0x7fad800783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 -- 172.21.15.26:0/3710972120 shutdown_connections 2024-04-23T18:17:18.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.062+0000 7fad87968640 1 -- 172.21.15.26:0/3710972120 wait complete. 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 Processor -- start 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 -- start start 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 0x7fad80198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 0x7fad80198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fad80112210 con 0x7fad8019d010 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad80112090 con 0x7fad80106190 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad87968640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fad80112390 con 0x7fad80101110 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58394/0 (socket says 172.21.15.26:58394) 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 -- 172.21.15.26:0/2791085270 learned_addr learned my addr 172.21.15.26:0/2791085270 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad84f03640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 0x7fad80198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85704640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 0x7fad80198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 -- 172.21.15.26:0/2791085270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 msgr2=0x7fad80198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 0x7fad80198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 -- 172.21.15.26:0/2791085270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 msgr2=0x7fad80198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 0x7fad80198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85f05640 1 -- 172.21.15.26:0/2791085270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8019db00 con 0x7fad8019d010 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad85704640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 0x7fad80198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.063+0000 7fad84f03640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 0x7fad80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad85f05640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e960 tx=0x7fad7800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78004d20 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad78005020 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8019ddf0 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad780116d0 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.064+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801a55e0 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.065+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad800691c0 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.066+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fad780051d0 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.067+0000 7fad767fc640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 0x7fad6c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.067+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fad780d0e90 con 0x7fad8019d010 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.067+0000 7fad85704640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 0x7fad6c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.068+0000 7fad85704640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 0x7fad6c0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad7c005d00 tx=0x7fad7c009550 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.072+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad78016030 con 0x7fad8019d010 2024-04-23T18:17:18.073 INFO:tasks.workunit.client.0.smithi026.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T18:17:18.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.299+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad80101ac0 con 0x7fad8019d010 2024-04-23T18:17:18.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.300+0000 7fad767fc640 1 -- 172.21.15.26:0/2791085270 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11884 (secure 0 0 0) 0x7fad7809d390 con 0x7fad8019d010 2024-04-23T18:17:18.295 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:18.295 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:08.076905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:17:18.295 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 22 2024-04-23T18:17:18.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 msgr2=0x7fad6c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 0x7fad6c0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad7c005d00 tx=0x7fad7c009550 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 msgr2=0x7fad8019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fad7800e960 tx=0x7fad7800ee20 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 shutdown_connections 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fad6c0a3db0 0x7fad6c0a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad80106190 0x7fad80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fad8019d010 0x7fad8019d4c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 --2- 172.21.15.26:0/2791085270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad80101110 0x7fad80198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 >> 172.21.15.26:0/2791085270 conn(0x7fad80077fb0 msgr2=0x7fad800ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 shutdown_connections 2024-04-23T18:17:18.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.302+0000 7fad87968640 1 -- 172.21.15.26:0/2791085270 wait complete. 2024-04-23T18:17:18.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:15:06.048360+0000', 'modified': '2024-04-23T18:17:08.076905+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24464, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24433': {'gid': 24433, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6835/2220145154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6834', 'nonce': 2220145154}, {'type': 'v1', 'addr': '172.21.15.131:6835', 'nonce': 2220145154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/3666168856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 3666168856}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 3666168856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6837/2204987461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6836', 'nonce': 2204987461}, {'type': 'v1', 'addr': '172.21.15.131:6837', 'nonce': 2204987461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.131:6839/4056718895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6838', 'nonce': 4056718895}, {'type': 'v1', 'addr': '172.21.15.131:6839', 'nonce': 4056718895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.110:6837/3430274697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 3430274697}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 3430274697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6839/3619972803', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 3619972803}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 3619972803}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24460, 24464, 24503]} 2024-04-23T18:17:18.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:17:18.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:17:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.456+0000 7fd1228cb640 1 -- 172.21.15.26:0/1194953151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 msgr2=0x7fd11c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.456+0000 7fd1228cb640 1 --2- 172.21.15.26:0/1194953151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c10ee30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd114009ca0 tx=0x7fd11402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 -- 172.21.15.26:0/1194953151 shutdown_connections 2024-04-23T18:17:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 --2- 172.21.15.26:0/1194953151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c069d00 0x7fd11c106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 --2- 172.21.15.26:0/1194953151 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c10ee30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 --2- 172.21.15.26:0/1194953151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 0x7fd11c069730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 -- 172.21.15.26:0/1194953151 >> 172.21.15.26:0/1194953151 conn(0x7fd11c077cb0 msgr2=0x7fd11c0780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 -- 172.21.15.26:0/1194953151 shutdown_connections 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 -- 172.21.15.26:0/1194953151 wait complete. 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 Processor -- start 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 -- start start 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 0x7fd11c1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.457+0000 7fd1228cb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 0x7fd11c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c1121d0 con 0x7fd11c107290 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c112050 con 0x7fd11c19a6b0 2024-04-23T18:17:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c112350 con 0x7fd11c069350 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58416/0 (socket says 172.21.15.26:58416) 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2438882714 learned_addr learned my addr 172.21.15.26:0/2438882714 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 0x7fd11c1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd120e68640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 0x7fd11c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2438882714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 msgr2=0x7fd11c1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 0x7fd11c1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2438882714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 msgr2=0x7fd11c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 0x7fd11c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2438882714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11c19b2b0 con 0x7fd11c107290 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd120e68640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 0x7fd11c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd11000c8c0 tx=0x7fd11000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1197fa640 1 -- 172.21.15.26:0/2438882714 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd110007bb0 con 0x7fd11c107290 2024-04-23T18:17:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1197fa640 1 -- 172.21.15.26:0/2438882714 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd110007d50 con 0x7fd11c107290 2024-04-23T18:17:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11c1a7340 con 0x7fd11c107290 2024-04-23T18:17:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1197fa640 1 -- 172.21.15.26:0/2438882714 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1100187c0 con 0x7fd11c107290 2024-04-23T18:17:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.458+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd11c1a7740 con 0x7fd11c107290 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.460+0000 7fd1197fa640 1 -- 172.21.15.26:0/2438882714 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fd11000fa20 con 0x7fd11c107290 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 Processor -- start 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 -- start start 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1197fa640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 0x7fd1040a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d65760 0x558e74d6f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d72220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558e74ccac50 con 0x558e74d64db0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558e74c67830 con 0x558e74d65760 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1197fa640 1 -- 172.21.15.26:0/2438882714 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd1100d09c0 con 0x7fd11c107290 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd1255d6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x558e74cc6c10 con 0x558e74d6fdd0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd120e68640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 0x7fd1040a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd11bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd11bfff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58420/0 (socket says 172.21.15.26:58420) 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.461+0000 7fd11bfff640 1 -- 172.21.15.26:0/485123969 learned_addr learned my addr 172.21.15.26:0/485123969 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd120e68640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d72220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d65760 0x558e74d6f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 -- 172.21.15.26:0/485123969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 msgr2=0x558e74d72220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d72220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 -- 172.21.15.26:0/485123969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d65760 msgr2=0x558e74d6f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d65760 0x558e74d6f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 -- 172.21.15.26:0/485123969 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e749dd2d0 con 0x558e74d64db0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd120e68640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d72220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd120e68640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 0x7fd1040a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd114006010 tx=0x7fd114031af0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd11bfff640 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00ac10 tx=0x7fd10c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cdb5367d33a868 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd1027fc640 1 -- 172.21.15.26:0/485123969 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c01d070 con 0x558e74d64db0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd1027fc640 1 -- 172.21.15.26:0/485123969 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd10c018af0 con 0x558e74d64db0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.462+0000 7fd1027fc640 1 -- 172.21.15.26:0/485123969 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c010a20 con 0x558e74d64db0 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- 172.21.15.26:0/485123969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 msgr2=0x558e74d65190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00ac10 tx=0x7fd10c00f790 comp rx=0 tx=0).stop 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- 172.21.15.26:0/485123969 shutdown_connections 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d65760 0x558e74d6f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d72220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 --2- 172.21.15.26:0/485123969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d64db0 0x558e74d65190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- 172.21.15.26:0/485123969 >> 172.21.15.26:0/485123969 conn(0x558e74d61070 msgr2=0x558e74d61450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- 172.21.15.26:0/485123969 shutdown_connections 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- 172.21.15.26:0/485123969 wait complete. 2024-04-23T18:17:18.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 -- start start 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.463+0000 7fd1255d6740 1 Processor -- start 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d6cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d6d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558e74d72d40 con 0x558e74d6fdd0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558e74c65470 con 0x558e74d64db0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x558e74d72ec0 con 0x558e74d65760 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 10 client.0 _ll_register_callbacks cb 0x558e74c8af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58436/0 (socket says 172.21.15.26:58436) 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 -- 172.21.15.26:0/2459240273 learned_addr learned my addr 172.21.15.26:0/2459240273 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d6cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d6d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 -- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 msgr2=0x558e74d6d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d6d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 -- 172.21.15.26:0/2459240273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 msgr2=0x558e74d6cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d6cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 -- 172.21.15.26:0/2459240273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e74aa76b0 con 0x558e74d6fdd0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d6d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d6cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd120e68640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098d0 tx=0x7fd11402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e856d8ad9b3470 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd0fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1140047c0 con 0x558e74d6fdd0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd10c018af0 con 0x558e74d6fdd0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.464+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd11402f660 con 0x558e74d6fdd0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd114057050 con 0x558e74d6fdd0 2024-04-23T18:17:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 10 client.15516 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558e74d7eeb0 con 0x558e74d6fdd0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd10c018af0 con 0x558e74d6fdd0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd114048070 con 0x558e74d6fdd0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 10 client.15516 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 10 client.15516 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 20 client.15516 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74d815c0 0x558e74d82a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 4 client.15516 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e74cd8670 con 0x558e74d815c0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.465+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74d815c0 0x558e74d82a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.466+0000 7fd0fb7fe640 10 client.15516 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.466+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74d815c0 0x558e74d82a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb45f5beeefd9a17 server_cookie=30c3515f8b1a196c in_seq=0 out_seq=0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/2459240273 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e74cd8670 con 0x558e74d815c0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd0fb7fe640 10 client.15516 handle_command_reply: tid=0 2024-04-23T18:17:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 client.15516 shutdown 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 msgr2=0x558e74d6d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098d0 tx=0x7fd11402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 shutdown_connections 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74d815c0 0x558e74d82a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d6d6b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d6d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2459240273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d6cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd0fb7fe640 0 client.15516 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 >> 172.21.15.26:0/2459240273 conn(0x558e74d61070 msgr2=0x558e74d66ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 shutdown_connections 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 1 -- 172.21.15.26:0/2459240273 wait complete. 2024-04-23T18:17:18.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.467+0000 7fd1255d6740 20 client.15516 trim_cache size 0 max 16384 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 Processor -- start 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 -- start start 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d67560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d6a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558e74d80d70 con 0x558e74d65760 2024-04-23T18:17:18.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558e74d81440 con 0x558e74d64db0 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd1255d6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x558e74d81b10 con 0x558e74d6fdd0 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58442/0 (socket says 172.21.15.26:58442) 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.485+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2795024142 learned_addr learned my addr 172.21.15.26:0/2795024142 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d67560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd120e68640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d6a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2795024142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 msgr2=0x558e74d6a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d6a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2795024142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 msgr2=0x558e74d67560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d67560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 -- 172.21.15.26:0/2795024142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e74d808e0 con 0x558e74d65760 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11bfff640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d67560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd120e68640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d6a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd110098c80 tx=0x7fd11009c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1ae267c1a1bd14 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd101ffb640 1 -- 172.21.15.26:0/2795024142 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11009c820 con 0x558e74d65760 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd101ffb640 1 -- 172.21.15.26:0/2795024142 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1100a0bd0 con 0x558e74d65760 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.486+0000 7fd101ffb640 1 -- 172.21.15.26:0/2795024142 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1100ce330 con 0x558e74d65760 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- 172.21.15.26:0/2795024142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 msgr2=0x558e74d69ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd110098c80 tx=0x7fd11009c420 comp rx=0 tx=0).stop 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- 172.21.15.26:0/2795024142 shutdown_connections 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d6fdd0 0x558e74d6a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d65760 0x558e74d69ed0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 --2- 172.21.15.26:0/2795024142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d67560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- 172.21.15.26:0/2795024142 >> 172.21.15.26:0/2795024142 conn(0x558e74d61070 msgr2=0x558e74d61450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- 172.21.15.26:0/2795024142 shutdown_connections 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- 172.21.15.26:0/2795024142 wait complete. 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 -- start start 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.487+0000 7fd1255d6740 1 Processor -- start 2024-04-23T18:17:18.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d695d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d69b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x558e74d828b0 con 0x558e74d6fdd0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558e74d821e0 con 0x558e74d64db0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x558e74d82f80 con 0x558e74d65760 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 10 client.0 _ll_register_callbacks cb 0x7fd114009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58450/0 (socket says 172.21.15.26:58450) 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 -- 172.21.15.26:0/1220037232 learned_addr learned my addr 172.21.15.26:0/1220037232 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd11bfff640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd11b7fe640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d69b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 -- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 msgr2=0x558e74d69b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d69b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 -- 172.21.15.26:0/1220037232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 msgr2=0x558e74d695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 -- 172.21.15.26:0/1220037232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e74d8a1c0 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd120e68640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd11402f350 tx=0x7fd114031c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6579879a85b9b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd0fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114004400 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd1140305c0 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1100a0bd0 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.488+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd114068050 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 10 client.15522 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558e74d8b470 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1100a0bd0 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd114048040 con 0x558e74d6fdd0 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 10 client.15522 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 10 client.15522 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:18.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 20 client.15522 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74debe10 0x558e74deab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 4 client.15522 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 --> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] -- command(tid 0: {"prefix": "status"}) -- 0x558e74cd24e0 con 0x558e74debe10 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.489+0000 7fd11bfff640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74debe10 0x558e74deab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd11bfff640 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74debe10 0x558e74deab20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99911905a3c1b6cd server_cookie=b1ce8265126e23c5 in_seq=0 out_seq=0 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd0fb7fe640 10 client.15522 ms_handle_connect on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd0fb7fe640 1 -- 172.21.15.26:0/1220037232 <== mds.2 v2:172.21.15.131:6838/4056718895 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558e74cd24e0 con 0x558e74debe10 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd0fb7fe640 10 client.15522 handle_command_reply: tid=0 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 client.15522 shutdown 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 msgr2=0x558e74d89c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd11402f350 tx=0x7fd114031c80 comp rx=0 tx=0).stop 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 shutdown_connections 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:6838/4056718895,v1:172.21.15.131:6839/4056718895] conn(0x558e74debe10 0x558e74deab20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x558e74d6fdd0 0x558e74d89c80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558e74d65760 0x558e74d69b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd0fb7fe640 0 client.15522 ms_handle_reset on v2:172.21.15.131:6838/4056718895 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 --2- 172.21.15.26:0/1220037232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558e74d64db0 0x558e74d695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 >> 172.21.15.26:0/1220037232 conn(0x558e74d61070 msgr2=0x558e74d6b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 shutdown_connections 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 1 -- 172.21.15.26:0/1220037232 wait complete. 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.490+0000 7fd1255d6740 20 client.15522 trim_cache size 0 max 16384 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 2, 2024-04-23T18:17:18.491 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24460, 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 127.79233100499999, 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 22, 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 146.389568967 2024-04-23T18:17:18.492 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 msgr2=0x7fd1040a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 0x7fd1040a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd114006010 tx=0x7fd114031af0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 msgr2=0x7fd11c1a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd11000c8c0 tx=0x7fd11000cd80 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 shutdown_connections 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd1040a3db0 0x7fd1040a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd11c19a6b0 0x7fd11c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd11c107290 0x7fd11c1a0ae0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 --2- 172.21.15.26:0/2438882714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd11c069350 0x7fd11c1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 >> 172.21.15.26:0/2438882714 conn(0x7fd11c077cb0 msgr2=0x7fd11c10d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 shutdown_connections 2024-04-23T18:17:18.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.491+0000 7fd1228cb640 1 -- 172.21.15.26:0/2438882714 wait complete. 2024-04-23T18:17:18.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 -- 172.21.15.26:0/4322702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde0101130 msgr2=0x7fcde0101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 --2- 172.21.15.26:0/4322702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde0101130 0x7fcde0101510 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009900 tx=0x7fcdd402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 -- 172.21.15.26:0/4322702 shutdown_connections 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 --2- 172.21.15.26:0/4322702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0101ae0 0x7fcde0105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 --2- 172.21.15.26:0/4322702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0106200 0x7fcde010ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 --2- 172.21.15.26:0/4322702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde0101130 0x7fcde0101510 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 -- 172.21.15.26:0/4322702 >> 172.21.15.26:0/4322702 conn(0x7fcde0077f80 msgr2=0x7fcde0078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 -- 172.21.15.26:0/4322702 shutdown_connections 2024-04-23T18:17:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.605+0000 7fcde666b640 1 -- 172.21.15.26:0/4322702 wait complete. 2024-04-23T18:17:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 Processor -- start 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 -- start start 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0101130 0x7fcde01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0106200 0x7fcde01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcde0111170 con 0x7fcde019a720 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcde0110ff0 con 0x7fcde0106200 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.606+0000 7fcde666b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fcde01112f0 con 0x7fcde0101130 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58476/0 (socket says 172.21.15.26:58476) 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 -- 172.21.15.26:0/2322310063 learned_addr learned my addr 172.21.15.26:0/2322310063 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcddffff640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0101130 0x7fcde01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0106200 0x7fcde01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 -- 172.21.15.26:0/2322310063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0101130 msgr2=0x7fcde01a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0101130 0x7fcde01a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 -- 172.21.15.26:0/2322310063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0106200 msgr2=0x7fcde01a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0106200 0x7fcde01a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 -- 172.21.15.26:0/2322310063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde019b320 con 0x7fcde019a720 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde4c08640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800eaa0 tx=0x7fcdd800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcddd7fa640 1 -- 172.21.15.26:0/2322310063 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd800cdb0 con 0x7fcde019a720 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcddd7fa640 1 -- 172.21.15.26:0/2322310063 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd8005020 con 0x7fcde019a720 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde01a7400 con 0x7fcde019a720 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcddd7fa640 1 -- 172.21.15.26:0/2322310063 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80116d0 con 0x7fcde019a720 2024-04-23T18:17:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.607+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde01a7860 con 0x7fcde019a720 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcddd7fa640 1 -- 172.21.15.26:0/2322310063 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fcdd80051d0 con 0x7fcde019a720 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 Processor -- start 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 -- start start 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x555571355c50 con 0x5555713f0760 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5555712f2830 con 0x5555713fadd0 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcde9380740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x555571351c10 con 0x5555713efdb0 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcddffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcddffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58490/0 (socket says 172.21.15.26:58490) 2024-04-23T18:17:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcddffff640 1 -- 172.21.15.26:0/4050663363 learned_addr learned my addr 172.21.15.26:0/4050663363 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.610+0000 7fcddd7fa640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 0x7fcdc80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcde4c08640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 0x7fcdc80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 -- 172.21.15.26:0/4050663363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 msgr2=0x5555713f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 -- 172.21.15.26:0/4050663363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 msgr2=0x5555713fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 -- 172.21.15.26:0/4050663363 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555710682d0 con 0x5555713f0760 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddd7fa640 1 -- 172.21.15.26:0/2322310063 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcdd80d0a30 con 0x7fcde019a720 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcddffff640 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009900 tx=0x7fcdd4032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e04cd7022a0068 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcde4c08640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 0x7fcdc80a62f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80ce150 tx=0x7fcdd80bd040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcdc67fc640 1 -- 172.21.15.26:0/4050663363 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4040070 con 0x5555713f0760 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcdc67fc640 1 -- 172.21.15.26:0/4050663363 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd40316a0 con 0x5555713f0760 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.611+0000 7fcdc67fc640 1 -- 172.21.15.26:0/4050663363 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4035040 con 0x5555713f0760 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- 172.21.15.26:0/4050663363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 msgr2=0x5555713fa800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009900 tx=0x7fcdd4032e60 comp rx=0 tx=0).stop 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- 172.21.15.26:0/4050663363 shutdown_connections 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713fa800 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 --2- 172.21.15.26:0/4050663363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- 172.21.15.26:0/4050663363 >> 172.21.15.26:0/4050663363 conn(0x5555713ec070 msgr2=0x5555713ec450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- 172.21.15.26:0/4050663363 shutdown_connections 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- 172.21.15.26:0/4050663363 wait complete. 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 -- start start 2024-04-23T18:17:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.612+0000 7fcde9380740 1 Processor -- start 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713f8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5555713fdce0 con 0x5555713efdb0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5555712f0470 con 0x5555713fadd0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5555713fde60 con 0x5555713f0760 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcde9380740 10 client.0 _ll_register_callbacks cb 0x555571315f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58500/0 (socket says 172.21.15.26:58500) 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 -- 172.21.15.26:0/3437451807 learned_addr learned my addr 172.21.15.26:0/3437451807 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddffff640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 -- 172.21.15.26:0/3437451807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 msgr2=0x5555713f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 -- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 msgr2=0x5555713f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 -- 172.21.15.26:0/3437451807 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555711326b0 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000ee40 tx=0x7fcdd000e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83826ccc41622968 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcdbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.613+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0005020 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcdd00051c0 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdd40316a0 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcdd0026050 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 10 client.15534 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55557140a020 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd40316a0 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcdd0011440 con 0x5555713efdb0 2024-04-23T18:17:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 10 client.15534 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 10 client.15534 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 20 client.15534 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x55557140c5e0 0x55557140da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 4 client.15534 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555571363670 con 0x55557140c5e0 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.614+0000 7fcddffff640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x55557140c5e0 0x55557140da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.615+0000 7fcdbf7fe640 10 client.15534 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.615+0000 7fcddffff640 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x55557140c5e0 0x55557140da50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b18c2db78dece0da server_cookie=1a70d146dfb4304c in_seq=0 out_seq=0 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.615+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/3437451807 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555571363670 con 0x55557140c5e0 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.615+0000 7fcdbf7fe640 10 client.15534 handle_command_reply: tid=0 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 client.15534 shutdown 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 msgr2=0x5555713f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000ee40 tx=0x7fcdd000e790 comp rx=0 tx=0).stop 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 shutdown_connections 2024-04-23T18:17:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x55557140c5e0 0x55557140da50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcdbf7fe640 0 client.15534 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 --2- 172.21.15.26:0/3437451807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713f7cc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 >> 172.21.15.26:0/3437451807 conn(0x5555713ec070 msgr2=0x5555713f1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 shutdown_connections 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 1 -- 172.21.15.26:0/3437451807 wait complete. 2024-04-23T18:17:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.616+0000 7fcde9380740 20 client.15534 trim_cache size 0 max 16384 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 Processor -- start 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 -- start start 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713ec450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55557140bd90 con 0x5555713efdb0 2024-04-23T18:17:18.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55557140c460 con 0x5555713fadd0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcde9380740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55557140cb30 con 0x5555713f0760 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcddf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcddf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58512/0 (socket says 172.21.15.26:58512) 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.634+0000 7fcddf7fe640 1 -- 172.21.15.26:0/1865941936 learned_addr learned my addr 172.21.15.26:0/1865941936 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 -- 172.21.15.26:0/1865941936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 msgr2=0x5555713ec450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddffff640 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713ec450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 -- 172.21.15.26:0/1865941936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 msgr2=0x5555713f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 -- 172.21.15.26:0/1865941936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555713ecde0 con 0x5555713efdb0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddffff640 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713ec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0010cd0 tx=0x7fcdd0010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751fb41494524bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcdc5ffb640 1 -- 172.21.15.26:0/1865941936 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0011e60 con 0x5555713efdb0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcdc5ffb640 1 -- 172.21.15.26:0/1865941936 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd000e3c0 con 0x5555713efdb0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.635+0000 7fcdc5ffb640 1 -- 172.21.15.26:0/1865941936 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00035f0 con 0x5555713efdb0 2024-04-23T18:17:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- 172.21.15.26:0/1865941936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 msgr2=0x5555713eb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0010cd0 tx=0x7fcdd0010d00 comp rx=0 tx=0).stop 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- 172.21.15.26:0/1865941936 shutdown_connections 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713f0760 0x5555713ec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- 172.21.15.26:0/1865941936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713efdb0 0x5555713eb950 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- 172.21.15.26:0/1865941936 >> 172.21.15.26:0/1865941936 conn(0x5555713ec070 msgr2=0x5555713f65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- 172.21.15.26:0/1865941936 shutdown_connections 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- 172.21.15.26:0/1865941936 wait complete. 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- start start 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 Processor -- start 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55557140d8d0 con 0x5555713f0760 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55557140d200 con 0x5555713fadd0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55557140dfa0 con 0x5555713efdb0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.636+0000 7fcde9380740 10 client.0 _ll_register_callbacks cb 0x7fcdd000e6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58514/0 (socket says 172.21.15.26:58514) 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 -- 172.21.15.26:0/365135711 learned_addr learned my addr 172.21.15.26:0/365135711 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 -- 172.21.15.26:0/365135711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 msgr2=0x5555713f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 -- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 msgr2=0x5555713f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 -- 172.21.15.26:0/365135711 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555571414020 con 0x5555713f0760 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcddffff640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x5555713f1370 tx=0x7fcdd4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba9187046677cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcdbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4035040 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcdd4004030 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55557135d4e0 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcdd4040040 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcde9380740 10 client.15540 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5555714728c0 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.637+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55557135d4e0 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcdd404d070 con 0x5555713f0760 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 10 client.15540 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 10 client.15540 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 20 client.15540 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x555571477030 0x555571475d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 4 client.15540 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 --> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] -- command(tid 0: {"prefix": "status"}) -- 0x5555712d99a0 con 0x555571477030 2024-04-23T18:17:18.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x555571477030 0x555571475d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcdbf7fe640 10 client.15540 ms_handle_connect on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcddf7fe640 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x555571477030 0x555571475d40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c9a47c777559e39 server_cookie=b4547d8b5cad72fd in_seq=0 out_seq=0 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcdbf7fe640 1 -- 172.21.15.26:0/365135711 <== mds.1 v2:172.21.15.110:6836/3430274697 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5555712d99a0 con 0x555571477030 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.638+0000 7fcdbf7fe640 10 client.15540 handle_command_reply: tid=0 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 client.15540 shutdown 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 msgr2=0x5555713f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x5555713f1370 tx=0x7fcdd4005970 comp rx=0 tx=0).stop 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 shutdown_connections 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:6836/3430274697,v1:172.21.15.110:6837/3430274697] conn(0x555571477030 0x555571475d40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5555713fadd0 0x5555713f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcdbf7fe640 0 client.15540 ms_handle_reset on v2:172.21.15.110:6836/3430274697 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5555713f0760 0x5555713f5140 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 --2- 172.21.15.26:0/365135711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5555713efdb0 0x5555713f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 >> 172.21.15.26:0/365135711 conn(0x5555713ec070 msgr2=0x555571406110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 shutdown_connections 2024-04-23T18:17:18.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 1 -- 172.21.15.26:0/365135711 wait complete. 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde9380740 20 client.15540 trim_cache size 0 max 16384 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 1, 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24464, 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 128.946427672, 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 22, 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:18.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:17:18.641 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 149.24045879100001 2024-04-23T18:17:18.641 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 msgr2=0x7fcdc80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 0x7fcdc80a62f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80ce150 tx=0x7fcdd80bd040 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 msgr2=0x7fcde019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800eaa0 tx=0x7fcdd800ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 shutdown_connections 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcdc80a3e30 0x7fcdc80a62f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcde0106200 0x7fcde01a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.639+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcde019a720 0x7fcde019aba0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.640+0000 7fcde666b640 1 --2- 172.21.15.26:0/2322310063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcde0101130 0x7fcde01a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.640+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 >> 172.21.15.26:0/2322310063 conn(0x7fcde0077f80 msgr2=0x7fcde01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.640+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 shutdown_connections 2024-04-23T18:17:18.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.640+0000 7fcde666b640 1 -- 172.21.15.26:0/2322310063 wait complete. 2024-04-23T18:17:18.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:18 smithi026 ceph-mon[29364]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:17:18.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:18 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/581118109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:18 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1016333371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:18 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2791085270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 -- 172.21.15.26:0/1057229893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 msgr2=0x7fc0d810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/1057229893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d810ee40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009ca0 tx=0x7fc0d002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 -- 172.21.15.26:0/1057229893 shutdown_connections 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/1057229893 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d810ee40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/1057229893 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d8102b50 0x7fc0d8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/1057229893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d81021a0 0x7fc0d8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 -- 172.21.15.26:0/1057229893 >> 172.21.15.26:0/1057229893 conn(0x7fc0d80fda20 msgr2=0x7fc0d80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 -- 172.21.15.26:0/1057229893 shutdown_connections 2024-04-23T18:17:18.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.754+0000 7fc0e06cf640 1 -- 172.21.15.26:0/1057229893 wait complete. 2024-04-23T18:17:18.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 Processor -- start 2024-04-23T18:17:18.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 -- start start 2024-04-23T18:17:18.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d81021a0 0x7fc0d8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d819cf40 0x7fc0d819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d81121e0 con 0x7fc0d8107270 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d8112060 con 0x7fc0d819cf40 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0e06cf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d8112360 con 0x7fc0d81021a0 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58538/0 (socket says 172.21.15.26:58538) 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/3172261085 learned_addr learned my addr 172.21.15.26:0/3172261085 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d819cf40 0x7fc0d819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0de46b640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d81021a0 0x7fc0d8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/3172261085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d81021a0 msgr2=0x7fc0d8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d81021a0 0x7fc0d8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/3172261085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d819cf40 msgr2=0x7fc0d819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d819cf40 0x7fc0d819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.755+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/3172261085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d819da30 con 0x7fc0d8107270 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800d740 tx=0x7fc0c800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0cf7fe640 1 -- 172.21.15.26:0/3172261085 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8017070 con 0x7fc0d8107270 2024-04-23T18:17:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0cf7fe640 1 -- 172.21.15.26:0/3172261085 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0c800de50 con 0x7fc0d8107270 2024-04-23T18:17:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d819dd20 con 0x7fc0d8107270 2024-04-23T18:17:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0cf7fe640 1 -- 172.21.15.26:0/3172261085 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8004ce0 con 0x7fc0d8107270 2024-04-23T18:17:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.756+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d81a54c0 con 0x7fc0d8107270 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0cf7fe640 1 -- 172.21.15.26:0/3172261085 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fc0c8004e80 con 0x7fc0d8107270 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 Processor -- start 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 -- start start 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 0x556d9f12e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f131220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556d9f089c50 con 0x556d9f123db0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d9f026830 con 0x556d9f124760 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556d9f085c10 con 0x556d9f12edd0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0dec6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0dec6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58546/0 (socket says 172.21.15.26:58546) 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0dec6c640 1 -- 172.21.15.26:0/1221098683 learned_addr learned my addr 172.21.15.26:0/1221098683 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0de46b640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 0x556d9f12e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f131220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.759+0000 7fc0cf7fe640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 0x7fc0c40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 -- 172.21.15.26:0/1221098683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 msgr2=0x556d9f131220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f131220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 -- 172.21.15.26:0/1221098683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 msgr2=0x556d9f12e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 0x556d9f12e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 -- 172.21.15.26:0/1221098683 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d9ed9c2d0 con 0x556d9f123db0 2024-04-23T18:17:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0de46b640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 0x556d9f12e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0cf7fe640 1 -- 172.21.15.26:0/3172261085 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc0c80d0350 con 0x7fc0d8107270 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 0x7fc0c40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005cc0 tx=0x7fc0d00099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0799a8b3d549d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0ccff9640 1 -- 172.21.15.26:0/1221098683 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d004d070 con 0x556d9f123db0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0ccff9640 1 -- 172.21.15.26:0/1221098683 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d0036040 con 0x556d9f123db0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.760+0000 7fc0ccff9640 1 -- 172.21.15.26:0/1221098683 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d002f860 con 0x556d9f123db0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- 172.21.15.26:0/1221098683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 msgr2=0x556d9f124190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 0x7fc0c40a6270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0031fd0 tx=0x7fc0d004b000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005cc0 tx=0x7fc0d00099c0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- 172.21.15.26:0/1221098683 shutdown_connections 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f124760 0x556d9f12e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f131220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/1221098683 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f124190 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- 172.21.15.26:0/1221098683 >> 172.21.15.26:0/1221098683 conn(0x556d9f120070 msgr2=0x556d9f120450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- 172.21.15.26:0/1221098683 shutdown_connections 2024-04-23T18:17:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- 172.21.15.26:0/1221098683 wait complete. 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 -- start start 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.761+0000 7fc0e33ed740 1 Processor -- start 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f123db0 0x556d9f12bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f12c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556d9f131d40 con 0x556d9f124760 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d9f024470 con 0x556d9f12edd0 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556d9f131ec0 con 0x556d9f123db0 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 10 client.0 _ll_register_callbacks cb 0x556d9f049f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58548/0 (socket says 172.21.15.26:58548) 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/476035347 learned_addr learned my addr 172.21.15.26:0/476035347 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0de46b640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f123db0 0x556d9f12bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/476035347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f123db0 msgr2=0x556d9f12bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f123db0 0x556d9f12bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 msgr2=0x556d9f12c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f12c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/476035347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d9ee666b0 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009610 tx=0x7fc0c809bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2fa56df4bec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8020040 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc0c80a0150 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d0036040 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc0c80a0350 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 10 client.15552 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.762+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556d9f13e080 con 0x556d9f124760 2024-04-23T18:17:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d0036040 con 0x556d9f124760 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc0c80cdd50 con 0x556d9f124760 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 10 client.15552 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 10 client.15552 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 20 client.15552 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f140660 0x556d9f141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 4 client.15552 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d9f097670 con 0x556d9f140660 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.763+0000 7fc0de46b640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f140660 0x556d9f141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0de46b640 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f140660 0x556d9f141ad0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=717deaa63d7ac44a server_cookie=59a14838f0956e03 in_seq=0 out_seq=0 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0c1ffb640 10 client.15552 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/476035347 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d9f097670 con 0x556d9f140660 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0c1ffb640 10 client.15552 handle_command_reply: tid=0 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 client.15552 shutdown 2024-04-23T18:17:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 msgr2=0x556d9f12c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009610 tx=0x7fc0c809bcf0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 shutdown_connections 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f140660 0x556d9f141ad0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f12c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f12c260 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0c1ffb640 0 client.15552 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/476035347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f123db0 0x556d9f12bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.764+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 >> 172.21.15.26:0/476035347 conn(0x556d9f120070 msgr2=0x556d9f125ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.765+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 shutdown_connections 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.765+0000 7fc0e33ed740 1 -- 172.21.15.26:0/476035347 wait complete. 2024-04-23T18:17:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.765+0000 7fc0e33ed740 20 client.15552 trim_cache size 0 max 16384 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 Processor -- start 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 -- start start 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f124760 0x556d9f12ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f128fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556d9f13fe10 con 0x556d9f123db0 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d9f1404e0 con 0x556d9f12edd0 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556d9f140bb0 con 0x556d9f124760 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58554/0 (socket says 172.21.15.26:58554) 2024-04-23T18:17:18.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 -- 172.21.15.26:0/4274682429 learned_addr learned my addr 172.21.15.26:0/4274682429 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 -- 172.21.15.26:0/4274682429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f124760 msgr2=0x556d9f12ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f124760 0x556d9f12ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 -- 172.21.15.26:0/4274682429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 msgr2=0x556d9f128fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f128fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.783+0000 7fc0de46b640 1 -- 172.21.15.26:0/4274682429 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d9f126b10 con 0x556d9f123db0 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0de46b640 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004700 tx=0x7fc0d400e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f902dd38720bad64 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0c3fff640 1 -- 172.21.15.26:0/4274682429 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4003e20 con 0x556d9f123db0 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0c3fff640 1 -- 172.21.15.26:0/4274682429 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d40054d0 con 0x556d9f123db0 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0c3fff640 1 -- 172.21.15.26:0/4274682429 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d401d070 con 0x556d9f123db0 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 -- 172.21.15.26:0/4274682429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 msgr2=0x556d9f12a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004700 tx=0x7fc0d400e750 comp rx=0 tx=0).stop 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 -- 172.21.15.26:0/4274682429 shutdown_connections 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f12edd0 0x556d9f128fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f124760 0x556d9f12ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/4274682429 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f123db0 0x556d9f12a5f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 -- 172.21.15.26:0/4274682429 >> 172.21.15.26:0/4274682429 conn(0x556d9f120070 msgr2=0x556d9f127020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 -- 172.21.15.26:0/4274682429 shutdown_connections 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.784+0000 7fc0e33ed740 1 -- 172.21.15.26:0/4274682429 wait complete. 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 -- start start 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 Processor -- start 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f123db0 0x556d9f128b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f1295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556d9f141950 con 0x556d9f124760 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d9f141280 con 0x556d9f123db0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556d9f142020 con 0x556d9f12edd0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0e33ed740 10 client.0 _ll_register_callbacks cb 0x7fc0c80cd8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58556/0 (socket says 172.21.15.26:58556) 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/50618254 learned_addr learned my addr 172.21.15.26:0/50618254 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0dec6c640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f1295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0de46b640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f123db0 0x556d9f128b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/50618254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 msgr2=0x556d9f1295e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f1295e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f123db0 msgr2=0x556d9f128b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f123db0 0x556d9f128b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 -- 172.21.15.26:0/50618254 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d9f148150 con 0x556d9f124760 2024-04-23T18:17:18.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0ddc6a640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc0c809c140 tx=0x7fc0c80a0380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf0006a779a126d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.785+0000 7fc0c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c809bb10 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc0c809b600 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d40054d0 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc0c80cded0 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 10 client.15558 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556d9f1a69e0 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d40054d0 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc0c8065020 con 0x556d9f124760 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 10 client.15558 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 10 client.15558 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 20 client.15558 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f1ab540 0x556d9f1aa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 4 client.15558 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "status"}) -- 0x556d9f0914e0 con 0x556d9f1ab540 2024-04-23T18:17:18.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.786+0000 7fc0de46b640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f1ab540 0x556d9f1aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0de46b640 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f1ab540 0x556d9f1aa250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=238ca9d74192c4e9 server_cookie=1c2471ccc8c52ac7 in_seq=0 out_seq=0 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0c1ffb640 10 client.15558 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0c1ffb640 1 -- 172.21.15.26:0/50618254 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556d9f0914e0 con 0x556d9f1ab540 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0c1ffb640 10 client.15558 handle_command_reply: tid=0 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 client.15558 shutdown 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 msgr2=0x556d9f1290a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc0c809c140 tx=0x7fc0c80a0380 comp rx=0 tx=0).stop 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 shutdown_connections 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556d9f1ab540 0x556d9f1aa250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556d9f12edd0 0x556d9f1295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556d9f124760 0x556d9f1290a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0c1ffb640 0 client.15558 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 --2- 172.21.15.26:0/50618254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d9f123db0 0x556d9f128b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 >> 172.21.15.26:0/50618254 conn(0x556d9f120070 msgr2=0x556d9f12a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 shutdown_connections 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 1 -- 172.21.15.26:0/50618254 wait complete. 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.787+0000 7fc0e33ed740 20 client.15558 trim_cache size 0 max 16384 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_fsid": "2a701120-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "whoami": 0, 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "id": 24503, 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "want_state": "up:active", 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "up:active", 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "fs_name": "cephfs", 2024-04-23T18:17:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "rank_uptime": 132.11433424500001, 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "mdsmap_epoch": 22, 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_epoch_barrier": 88, 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 138.12763282899999 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:18.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 msgr2=0x7fc0c40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 0x7fc0c40a6270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0031fd0 tx=0x7fc0d004b000 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 msgr2=0x7fc0d8198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800d740 tx=0x7fc0c800dc00 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 shutdown_connections 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0c40a3db0 0x7fc0c40a6270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d819cf40 0x7fc0d819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0d8107270 0x7fc0d8198bc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 --2- 172.21.15.26:0/3172261085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0d81021a0 0x7fc0d8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 >> 172.21.15.26:0/3172261085 conn(0x7fc0d80fda20 msgr2=0x7fc0d80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 shutdown_connections 2024-04-23T18:17:18.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:18.788+0000 7fc0e06cf640 1 -- 172.21.15.26:0/3172261085 wait complete. 2024-04-23T18:17:18.793 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-23T18:17:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:18 smithi110 ceph-mon[32585]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:17:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:18 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/581118109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:18 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1016333371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:18 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2791085270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:18 smithi131 ceph-mon[33041]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:17:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:18 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/581118109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:18 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1016333371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:18.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:18 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2791085270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:20.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:20 smithi110 ceph-mon[32585]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:17:20.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:20 smithi026 ceph-mon[29364]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:17:20.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:20 smithi131 ceph-mon[33041]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:17:22.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:22 smithi110 ceph-mon[32585]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:17:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:22 smithi026 ceph-mon[29364]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:17:22.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:22 smithi131 ceph-mon[33041]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:17:24.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:24 smithi110 ceph-mon[32585]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T18:17:24.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:24 smithi026 ceph-mon[29364]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T18:17:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1424295463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 msgr2=0x7f3fb810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1424295463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb810dd40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009ca0 tx=0x7f3fb002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1424295463 shutdown_connections 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1424295463 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb810dd40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1424295463 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb8101aa0 0x7f3fb8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1424295463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb81010f0 0x7f3fb81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1424295463 >> 172.21.15.26:0/1424295463 conn(0x7f3fb8077fd0 msgr2=0x7f3fb80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1424295463 shutdown_connections 2024-04-23T18:17:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.970+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1424295463 wait complete. 2024-04-23T18:17:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 Processor -- start 2024-04-23T18:17:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- start start 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 0x7f3fb8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb819cfe0 0x7f3fb819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81110e0 con 0x7f3fb8106170 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb8110f60 con 0x7f3fb819cfe0 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb8111260 con 0x7f3fb81010f0 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58578/0 (socket says 172.21.15.26:58578) 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/1904730371 learned_addr learned my addr 172.21.15.26:0/1904730371 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb819cfe0 0x7f3fb819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 0x7f3fb8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/1904730371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 msgr2=0x7f3fb8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 0x7f3fb8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/1904730371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb819cfe0 msgr2=0x7f3fb819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb819cfe0 0x7f3fb819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/1904730371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb819dad0 con 0x7f3fb8106170 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 0x7f3fb8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00d740 tx=0x7f3fac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb4ff9640 1 -- 172.21.15.26:0/1904730371 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac017070 con 0x7f3fb8106170 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb4ff9640 1 -- 172.21.15.26:0/1904730371 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fac00de50 con 0x7f3fb8106170 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb819ddc0 con 0x7f3fb8106170 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fb4ff9640 1 -- 172.21.15.26:0/1904730371 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac004ce0 con 0x7f3fb8106170 2024-04-23T18:17:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.972+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb81a5560 con 0x7f3fb8106170 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.974+0000 7f3fb4ff9640 1 -- 172.21.15.26:0/1904730371 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3fac004e80 con 0x7f3fb8106170 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 Processor -- start 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 -- start start 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb4ff9640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 0x7f3fa00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 0x55e3bbd2ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbc90970 con 0x55e3bbd2bf40 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbc2d4a0 con 0x55e3bbd2b620 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbc8c930 con 0x55e3bbd36800 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 0x7f3fa00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb4ff9640 1 -- 172.21.15.26:0/1904730371 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3fac0d02a0 con 0x7f3fb8106170 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 0x55e3bbd2ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58592/0 (socket says 172.21.15.26:58592) 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.975+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/2715091847 learned_addr learned my addr 172.21.15.26:0/2715091847 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2715091847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 msgr2=0x55e3bbd38bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2715091847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 msgr2=0x55e3bbd362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2715091847 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3bb9a3210 con 0x55e3bbd2b620 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 0x7f3fa00a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006010 tx=0x7f3fb0005f60 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 0x55e3bbd2ba00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005fd0 tx=0x7f3fa800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c644169428f51b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3f9dffb640 1 -- 172.21.15.26:0/2715091847 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8004030 con 0x55e3bbd2b620 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3f9dffb640 1 -- 172.21.15.26:0/2715091847 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa80054d0 con 0x55e3bbd2b620 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.976+0000 7f3f9dffb640 1 -- 172.21.15.26:0/2715091847 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8004b20 con 0x55e3bbd2b620 2024-04-23T18:17:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2715091847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 msgr2=0x55e3bbd2ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 0x55e3bbd2ba00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005fd0 tx=0x7f3fa800f730 comp rx=0 tx=0).stop 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2715091847 shutdown_connections 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd38bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2715091847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2b620 0x55e3bbd2ba00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2715091847 >> 172.21.15.26:0/2715091847 conn(0x55e3bbd27b30 msgr2=0x55e3bbd27f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2715091847 shutdown_connections 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2715091847 wait complete. 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 -- start start 2024-04-23T18:17:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.977+0000 7f3fc0c79740 1 Processor -- start 2024-04-23T18:17:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2b620 0x55e3bbd36130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbd39430 con 0x55e3bbd36800 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbc2b0b0 con 0x55e3bbd2bf40 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e3bbd395b0 con 0x55e3bbd2b620 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fc0c79740 10 client.0 _ll_register_callbacks cb 0x55e3bbc18d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58596/0 (socket says 172.21.15.26:58596) 2024-04-23T18:17:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:46442/0 (socket says 172.21.15.26:46442) 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2b620 0x55e3bbd36130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb7fff640 1 -- 172.21.15.26:0/3285401562 learned_addr learned my addr 172.21.15.26:0/3285401562 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/3285401562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2b620 msgr2=0x55e3bbd36130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2b620 0x55e3bbd36130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/3285401562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 msgr2=0x55e3bbd33960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 -- 172.21.15.26:0/3285401562 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3bba6f620 con 0x55e3bbd2bf40 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fac09be00 tx=0x7f3fac09c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2cbeb3e8855c028 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3f96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.978+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0cf040 con 0x55e3bbd2bf40 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3fac009c30 con 0x55e3bbd2bf40 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fa80054d0 con 0x55e3bbd2bf40 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3fac020040 con 0x55e3bbd2bf40 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 10 client.25118 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e3bbd45390 con 0x55e3bbd2bf40 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa80054d0 con 0x55e3bbd2bf40 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3fac0cdc10 con 0x55e3bbd2bf40 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 10 client.25118 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 10 client.25118 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 20 client.25118 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbd468a0 0x55e3bbd48cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 4 client.25118 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3bbc9e390 con 0x55e3bbd468a0 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.979+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbd468a0 0x55e3bbd48cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.980+0000 7f3f96ffd640 10 client.25118 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.980+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbd468a0 0x55e3bbd48cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eaaacde13cdf00e server_cookie=852f7b984bfad618 in_seq=0 out_seq=0 2024-04-23T18:17:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.980+0000 7f3f96ffd640 1 -- 172.21.15.26:0/3285401562 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3bbc9e390 con 0x55e3bbd468a0 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.980+0000 7f3f96ffd640 10 client.25118 handle_command_reply: tid=0 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.980+0000 7f3fc0c79740 1 client.25118 shutdown 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 msgr2=0x55e3bbd33420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fac09be00 tx=0x7f3fac09c1c0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 shutdown_connections 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbd468a0 0x55e3bbd48cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd33420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3f96ffd640 0 client.25118 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd36800 0x55e3bbd33960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/3285401562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2b620 0x55e3bbd36130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 >> 172.21.15.26:0/3285401562 conn(0x55e3bbd27b30 msgr2=0x55e3bbd2d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 shutdown_connections 2024-04-23T18:17:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 1 -- 172.21.15.26:0/3285401562 wait complete. 2024-04-23T18:17:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.981+0000 7f3fc0c79740 20 client.25118 trim_cache size 0 max 16384 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 Processor -- start 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 -- start start 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd26b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 0x55e3bbd4e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81b0350 con 0x55e3bbd2b620 2024-04-23T18:17:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81adc20 con 0x55e3bbd36800 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81aba70 con 0x55e3bbd2bf40 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58604/0 (socket says 172.21.15.26:58604) 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2612630590 learned_addr learned my addr 172.21.15.26:0/2612630590 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 0x55e3bbd4e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd26b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2612630590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2bf40 msgr2=0x55e3bbd26b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd26b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2612630590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 msgr2=0x55e3bbd4e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 0x55e3bbd4e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 -- 172.21.15.26:0/2612630590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3bbd31990 con 0x55e3bbd2b620 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 0x55e3bbd4e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.997+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005fd0 tx=0x7f3fa801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5901ff5dc416ed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3f9d7fa640 1 -- 172.21.15.26:0/2612630590 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8030070 con 0x55e3bbd2b620 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3f9d7fa640 1 -- 172.21.15.26:0/2612630590 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa802b660 con 0x55e3bbd2b620 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3f9d7fa640 1 -- 172.21.15.26:0/2612630590 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa801ba00 con 0x55e3bbd2b620 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2612630590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 msgr2=0x55e3bbd422f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8005fd0 tx=0x7f3fa801f040 comp rx=0 tx=0).stop 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2612630590 shutdown_connections 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd36800 0x55e3bbd4e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd26b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/2612630590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd422f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2612630590 >> 172.21.15.26:0/2612630590 conn(0x55e3bbd27b30 msgr2=0x55e3bbd26420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.998+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2612630590 shutdown_connections 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 -- 172.21.15.26:0/2612630590 wait complete. 2024-04-23T18:17:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 -- start start 2024-04-23T18:17:24.996 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:24 smithi131 ceph-mon[33041]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T18:17:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 Processor -- start 2024-04-23T18:17:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd3c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd3c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81ac0e0 con 0x55e3bbd2b620 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81ad550 con 0x55e3bbd2bf40 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb81ac7b0 con 0x55e3bbd36800 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fc0c79740 10 client.0 _ll_register_callbacks cb 0x7f3fac09c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fb77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58612/0 (socket says 172.21.15.26:58612) 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:24.999+0000 7f3fb77fe640 1 -- 172.21.15.26:0/1527136764 learned_addr learned my addr 172.21.15.26:0/1527136764 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd3c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb7fff640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd3c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 -- 172.21.15.26:0/1527136764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 msgr2=0x55e3bbd3c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd3c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 -- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 msgr2=0x55e3bbd3c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd3c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 -- 172.21.15.26:0/1527136764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3bbd3ced0 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fb77fe640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800f770 tx=0x7f3fa800b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1ff679065ff929 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3f96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa803b060 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3fa802b1b0 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fa802b660 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3fa8030070 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fc0c79740 10 client.15576 fetch_fsmap learned FSMap version 22 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.000+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e3bbdacf50 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa802b660 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3fa801ba40 con 0x55e3bbd2b620 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 10 client.15576 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 10 client.15576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 20 client.15576 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbdad410 0x55e3bbdaf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 4 client.15576 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e3bbd33cf0 con 0x55e3bbdad410 2024-04-23T18:17:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbdad410 0x55e3bbdaf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.001+0000 7f3f96ffd640 10 client.15576 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.002+0000 7f3fb6ffd640 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbdad410 0x55e3bbdaf800 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=865b9b903717fcfc server_cookie=1d177b406cdc01a3 in_seq=0 out_seq=0 2024-04-23T18:17:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3f96ffd640 1 -- 172.21.15.26:0/1527136764 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e3bbd33cf0 con 0x55e3bbdad410 2024-04-23T18:17:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3f96ffd640 10 client.15576 handle_command_reply: tid=0 2024-04-23T18:17:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 client.15576 shutdown 2024-04-23T18:17:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 msgr2=0x55e3bbd3ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800f770 tx=0x7f3fa800b4d0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 shutdown_connections 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e3bbdad410 0x55e3bbdaf800 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e3bbd2bf40 0x55e3bbd3c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3f96ffd640 0 client.15576 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3bbd36800 0x55e3bbd3c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 --2- 172.21.15.26:0/1527136764 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e3bbd2b620 0x55e3bbd3ecb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 >> 172.21.15.26:0/1527136764 conn(0x55e3bbd27b30 msgr2=0x55e3bbd26cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 shutdown_connections 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.012+0000 7f3fc0c79740 1 -- 172.21.15.26:0/1527136764 wait complete. 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fc0c79740 20 client.15576 trim_cache size 0 max 16384 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:17:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:17:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:17:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:17:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 msgr2=0x7f3fa00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 0x7f3fa00a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006010 tx=0x7f3fb0005f60 comp rx=0 tx=0).stop 2024-04-23T18:17:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 msgr2=0x7f3fb8198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00d740 tx=0x7f3fac00dc00 comp rx=0 tx=0).stop 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 shutdown_connections 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3fa00a3e60 0x7f3fa00a6320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb819cfe0 0x7f3fb819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3fb8106170 0x7f3fb8198c60 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 --2- 172.21.15.26:0/1904730371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fb81010f0 0x7f3fb8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 >> 172.21.15.26:0/1904730371 conn(0x7f3fb8077fd0 msgr2=0x7f3fb8104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 shutdown_connections 2024-04-23T18:17:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:25.013+0000 7f3fbdf5d640 1 -- 172.21.15.26:0/1904730371 wait complete. 2024-04-23T18:17:26.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:26 smithi110 ceph-mon[32585]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:17:26.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:26 smithi026 ceph-mon[29364]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:17:26.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:26 smithi131 ceph-mon[33041]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:17:28.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:28 smithi110 ceph-mon[32585]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:17:28.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:28 smithi026 ceph-mon[29364]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:17:28.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:28 smithi131 ceph-mon[33041]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:17:30.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:30 smithi110 ceph-mon[32585]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-23T18:17:30.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:30 smithi026 ceph-mon[29364]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-23T18:17:30.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:30 smithi131 ceph-mon[33041]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-23T18:17:32.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:32 smithi110 ceph-mon[32585]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-23T18:17:32.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:32 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:32.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:32 smithi026 ceph-mon[29364]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-23T18:17:32.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:32 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:32.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:32 smithi131 ceph-mon[33041]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-23T18:17:32.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:32 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:33.794 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.5 2024-04-23T18:17:33.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 -- 172.21.15.26:0/506346818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d41031e0 msgr2=0x7fc7d41035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 --2- 172.21.15.26:0/506346818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d41031e0 0x7fc7d41035c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009900 tx=0x7fc7c402a510 comp rx=0 tx=0).stop 2024-04-23T18:17:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 -- 172.21.15.26:0/506346818 shutdown_connections 2024-04-23T18:17:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 --2- 172.21.15.26:0/506346818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d410c520 0x7fc7d410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 --2- 172.21.15.26:0/506346818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d4103b00 0x7fc7d410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 --2- 172.21.15.26:0/506346818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d41031e0 0x7fc7d41035c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 -- 172.21.15.26:0/506346818 >> 172.21.15.26:0/506346818 conn(0x7fc7d4077f50 msgr2=0x7fc7d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 -- 172.21.15.26:0/506346818 shutdown_connections 2024-04-23T18:17:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.923+0000 7fc7daa20640 1 -- 172.21.15.26:0/506346818 wait complete. 2024-04-23T18:17:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.924+0000 7fc7daa20640 1 Processor -- start 2024-04-23T18:17:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.924+0000 7fc7daa20640 1 -- start start 2024-04-23T18:17:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 0x7fc7d4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d410c520 0x7fc7d419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112190 con 0x7fc7d4103b00 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112010 con 0x7fc7d41031e0 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7daa20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112310 con 0x7fc7d410c520 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48900/0 (socket says 172.21.15.26:48900) 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 -- 172.21.15.26:0/1721668845 learned_addr learned my addr 172.21.15.26:0/1721668845 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d3fff640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 0x7fc7d4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d8fbd640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d410c520 0x7fc7d419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 -- 172.21.15.26:0/1721668845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d410c520 msgr2=0x7fc7d419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d410c520 0x7fc7d419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 -- 172.21.15.26:0/1721668845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 msgr2=0x7fc7d4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 0x7fc7d4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 -- 172.21.15.26:0/1721668845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d419d720 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d3fff640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 0x7fc7d4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.925+0000 7fc7d37fe640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800d950 tx=0x7fc7c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.926+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8017070 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.926+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d419da10 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.926+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7c8009c30 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.926+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800f620 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.926+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a52a0 con 0x7fc7d4103b00 2024-04-23T18:17:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.927+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d4068d70 con 0x7fc7d4103b00 2024-04-23T18:17:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.929+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fc7c8005030 con 0x7fc7d4103b00 2024-04-23T18:17:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.929+0000 7fc7d17fa640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 0x7fc7bc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.930+0000 7fc7d3fff640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 0x7fc7bc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.930+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc7c80d0660 con 0x7fc7d4103b00 2024-04-23T18:17:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.930+0000 7fc7d3fff640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 0x7fc7bc0a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4031440 tx=0x7fc7c4005a20 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:33.933+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7c809cbc0 con 0x7fc7d4103b00 2024-04-23T18:17:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.191+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7d4199a80 con 0x7fc7d4103b00 2024-04-23T18:17:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.192+0000 7fc7d17fa640 1 -- 172.21.15.26:0/1721668845 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11885 (secure 0 0 0) 0x7fc7d4199a80 con 0x7fc7d4103b00 2024-04-23T18:17:34.185 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:34.185 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:30.666563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24460,24464]},"id":1}]} 2024-04-23T18:17:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 23 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 msgr2=0x7fc7bc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 0x7fc7bc0a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4031440 tx=0x7fc7c4005a20 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 msgr2=0x7fc7d4198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800d950 tx=0x7fc7c800de10 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 shutdown_connections 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7bc0a3db0 0x7fc7bc0a6270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc7d410c520 0x7fc7d419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc7d4103b00 0x7fc7d4198c70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 --2- 172.21.15.26:0/1721668845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7d41031e0 0x7fc7d4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 >> 172.21.15.26:0/1721668845 conn(0x7fc7d4077f50 msgr2=0x7fc7d40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 shutdown_connections 2024-04-23T18:17:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.194+0000 7fc7daa20640 1 -- 172.21.15.26:0/1721668845 wait complete. 2024-04-23T18:17:34.200 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 74.5 --await-for 42.0' 2024-04-23T18:17:34.200 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 74.5 --await-for 42.0 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 -- 172.21.15.26:0/336026124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 msgr2=0x7f79fc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/336026124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc101480 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009900 tx=0x7f79f802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 -- 172.21.15.26:0/336026124 shutdown_connections 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/336026124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 0x7f79fc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/336026124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc101a50 0x7f79fc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/336026124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc101480 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 -- 172.21.15.26:0/336026124 >> 172.21.15.26:0/336026124 conn(0x7f79fc077f50 msgr2=0x7f79fc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 -- 172.21.15.26:0/336026124 shutdown_connections 2024-04-23T18:17:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.325+0000 7f7a02a8a640 1 -- 172.21.15.26:0/336026124 wait complete. 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 Processor -- start 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 -- start start 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 0x7f79fc103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 0x7f79fc1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1110b0 con 0x7f79fc1010a0 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc110f30 con 0x7f79fc106170 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a02a8a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc111230 con 0x7f79fc1035b0 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a00826640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a00826640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48916/0 (socket says 172.21.15.26:48916) 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.326+0000 7f7a00826640 1 -- 172.21.15.26:0/2485360695 learned_addr learned my addr 172.21.15.26:0/2485360695 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 0x7f79fc103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a01027640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 0x7f79fc1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 -- 172.21.15.26:0/2485360695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 msgr2=0x7f79fc1a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 0x7f79fc1a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 -- 172.21.15.26:0/2485360695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 msgr2=0x7f79fc103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 0x7f79fc103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 -- 172.21.15.26:0/2485360695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc1a7010 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a01027640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 0x7f79fc1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a00826640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79f8005940 tx=0x7f79f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f79f1ffb640 1 -- 172.21.15.26:0/2485360695 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f803f030 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f79f1ffb640 1 -- 172.21.15.26:0/2485360695 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f8003aa0 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79fc1a72a0 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f79f1ffb640 1 -- 172.21.15.26:0/2485360695 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8035070 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.327+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79fc1a77e0 con 0x7f79fc1010a0 2024-04-23T18:17:34.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.330+0000 7f79f1ffb640 1 -- 172.21.15.26:0/2485360695 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f79f8042030 con 0x7f79fc1010a0 2024-04-23T18:17:34.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f79f1ffb640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 0x7f79e40a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 Processor -- start 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 -- start start 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 0x7f79e40a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f79f1ffb640 1 -- 172.21.15.26:0/2485360695 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f79f80f0dc0 con 0x7f79fc1010a0 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d07480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d079c0 0x564028d11b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564028c6c370 con 0x564028d12090 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564028c092e0 con 0x564028d079c0 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a057a4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564028c68330 con 0x564028d070a0 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a01027640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a01027640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48926/0 (socket says 172.21.15.26:48926) 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.331+0000 7f7a01027640 1 -- 172.21.15.26:0/2552318806 learned_addr learned my addr 172.21.15.26:0/2552318806 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:34.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a00826640 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d07480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d079c0 0x564028d11b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 0x7f79e40a6320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79fc1045c0 tx=0x7f79ec005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 -- 172.21.15.26:0/2552318806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 msgr2=0x564028d07480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d07480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 -- 172.21.15.26:0/2552318806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d079c0 msgr2=0x564028d11b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d079c0 0x564028d11b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 -- 172.21.15.26:0/2552318806 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564028a4ad80 con 0x564028d12090 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f7a01027640 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009e40 tx=0x7f79f400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fde46797451f48e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f79e2ffd640 1 -- 172.21.15.26:0/2552318806 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4004d20 con 0x564028d12090 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f79e2ffd640 1 -- 172.21.15.26:0/2552318806 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f40054d0 con 0x564028d12090 2024-04-23T18:17:34.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.332+0000 7f79e2ffd640 1 -- 172.21.15.26:0/2552318806 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011800 con 0x564028d12090 2024-04-23T18:17:34.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- 172.21.15.26:0/2552318806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 msgr2=0x564028d14480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009e40 tx=0x7f79f400f730 comp rx=0 tx=0).stop 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- 172.21.15.26:0/2552318806 shutdown_connections 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d079c0 0x564028d11b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d14480 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2552318806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d07480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- 172.21.15.26:0/2552318806 >> 172.21.15.26:0/2552318806 conn(0x564028d033c0 msgr2=0x564028d037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- 172.21.15.26:0/2552318806 shutdown_connections 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- 172.21.15.26:0/2552318806 wait complete. 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.333+0000 7f7a057a4740 1 -- start start 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 Processor -- start 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d0f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564028d14e80 con 0x564028d079c0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564028c06fb0 con 0x564028d12090 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564028d15000 con 0x564028d070a0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a057a4740 10 client.0 _ll_register_callbacks cb 0x564028c069a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48930/0 (socket says 172.21.15.26:48930) 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f79f3fff640 1 -- 172.21.15.26:0/2300261757 learned_addr learned my addr 172.21.15.26:0/2300261757 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a00826640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d0f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.334+0000 7f7a01027640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 -- 172.21.15.26:0/2300261757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 msgr2=0x564028d0f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d0f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 -- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 msgr2=0x564028d0fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 -- 172.21.15.26:0/2300261757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564028aab510 con 0x564028d079c0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f7a00826640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d0f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79f3fff640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f79ec012bc0 tx=0x7f79ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64d63a78c60f13a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec01fdd0 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f79ec01d450 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f40054d0 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f79ec01d650 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f7a057a4740 10 client.15594 fetch_fsmap learned FSMap version 23 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56402893e670 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.335+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f40054d0 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 23) ==== 8342+0+0 (secure 0 0 0) 0x7f79ec026070 con 0x564028d079c0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 10 client.15594 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 10 client.15594 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 20 client.15594 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d22680 0x564028d24b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 4 client.15594 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564028c79d90 con 0x564028d22680 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a00826640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d22680 0x564028d24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f79dbfff640 10 client.15594 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.336+0000 7f7a00826640 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d22680 0x564028d24b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217c34a8bcda17da server_cookie=4129816293361769 in_seq=0 out_seq=0 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f79dbfff640 1 -- 172.21.15.26:0/2300261757 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564028c79d90 con 0x564028d22680 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f79dbfff640 10 client.15594 handle_command_reply: tid=0 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 client.15594 shutdown 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 msgr2=0x564028d0f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f79ec012bc0 tx=0x7f79ec004730 comp rx=0 tx=0).stop 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 shutdown_connections 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d22680 0x564028d24b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d079c0 0x564028d0f5c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 --2- 172.21.15.26:0/2300261757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d070a0 0x564028d0f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f79dbfff640 0 client.15594 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.337+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 >> 172.21.15.26:0/2300261757 conn(0x564028d033c0 msgr2=0x564028d091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.338+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 shutdown_connections 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.338+0000 7f7a057a4740 1 -- 172.21.15.26:0/2300261757 wait complete. 2024-04-23T18:17:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.338+0000 7f7a057a4740 20 client.15594 trim_cache size 0 max 16384 2024-04-23T18:17:34.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 Processor -- start 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 -- start start 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b2660 con 0x564028d070a0 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1aff30 con 0x564028d12090 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a057a4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1add80 con 0x564028d079c0 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a00826640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a00826640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48936/0 (socket says 172.21.15.26:48936) 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a00826640 1 -- 172.21.15.26:0/1116020596 learned_addr learned my addr 172.21.15.26:0/1116020596 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f79f3fff640 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.356+0000 7f7a01027640 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 -- 172.21.15.26:0/1116020596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 msgr2=0x564028d1dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d1dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 -- 172.21.15.26:0/1116020596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 msgr2=0x564028d0f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 -- 172.21.15.26:0/1116020596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564028d077b0 con 0x564028d070a0 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f7a00826640 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f79f80b95f0 tx=0x7f79f8031950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7113dac1dca155 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f79e27fc640 1 -- 172.21.15.26:0/1116020596 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8049040 con 0x564028d070a0 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f79e27fc640 1 -- 172.21.15.26:0/1116020596 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f80d5070 con 0x564028d070a0 2024-04-23T18:17:34.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.357+0000 7f79e27fc640 1 -- 172.21.15.26:0/1116020596 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8003c50 con 0x564028d070a0 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- 172.21.15.26:0/1116020596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 msgr2=0x564028d02d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f79f80b95f0 tx=0x7f79f8031950 comp rx=0 tx=0).stop 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- 172.21.15.26:0/1116020596 shutdown_connections 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d12090 0x564028d0f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d1dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 --2- 172.21.15.26:0/1116020596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d070a0 0x564028d02d80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- 172.21.15.26:0/1116020596 >> 172.21.15.26:0/1116020596 conn(0x564028d033c0 msgr2=0x564028d0f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- 172.21.15.26:0/1116020596 shutdown_connections 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- 172.21.15.26:0/1116020596 wait complete. 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 -- start start 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.358+0000 7f7a057a4740 1 Processor -- start 2024-04-23T18:17:34.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d070a0 0x564028d1a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d0b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1ae3f0 con 0x564028d12090 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1af860 con 0x564028d070a0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1aeac0 con 0x564028d079c0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 10 client.0 _ll_register_callbacks cb 0x7f79ec0128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48940/0 (socket says 172.21.15.26:48940) 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 -- 172.21.15.26:0/464769391 learned_addr learned my addr 172.21.15.26:0/464769391 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f79f3fff640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d0b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a00826640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d070a0 0x564028d1a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 -- 172.21.15.26:0/464769391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 msgr2=0x564028d0b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d0b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 -- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d070a0 msgr2=0x564028d1a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d070a0 0x564028d1a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 -- 172.21.15.26:0/464769391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564028d0c480 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a01027640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f79f40099f0 tx=0x7f79f4004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0a82477bb6aac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f79dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f40118e0 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f79f4011a80 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.359+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f80d5070 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f79f4011c80 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 10 client.15600 fetch_fsmap learned FSMap version 23 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564028d88b30 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f80d5070 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 23) ==== 8342+0+0 (secure 0 0 0) 0x7f79f4019430 con 0x564028d12090 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 10 client.15600 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 10 client.15600 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 20 client.15600 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:34.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d891b0 0x564028d8b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 4 client.15600 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.5, "await_for": 42.0}] 2024-04-23T18:17:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.5, "await_for": 42.0}) -- 0x564028c73c00 con 0x564028d891b0 2024-04-23T18:17:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.360+0000 7f7a00826640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d891b0 0x564028d8b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.361+0000 7f79dbfff640 10 client.15600 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.361+0000 7f7a00826640 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d891b0 0x564028d8b5d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19ba864de2a51897 server_cookie=1b866894bc5ed7f5 in_seq=0 out_seq=0 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f79dbfff640 1 -- 172.21.15.26:0/464769391 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x564028c73c00 con 0x564028d891b0 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f79dbfff640 10 client.15600 handle_command_reply: tid=0 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 client.15600 shutdown 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 msgr2=0x564028d0be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f79f40099f0 tx=0x7f79f4004bf0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 shutdown_connections 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564028d891b0 0x564028d8b5d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564028d12090 0x564028d0be10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564028d079c0 0x564028d0b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 --2- 172.21.15.26:0/464769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564028d070a0 0x564028d1a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f79dbfff640 0 client.15600 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 >> 172.21.15.26:0/464769391 conn(0x564028d033c0 msgr2=0x564028d0f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 shutdown_connections 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 1 -- 172.21.15.26:0/464769391 wait complete. 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.368+0000 7f7a057a4740 20 client.15600 trim_cache size 0 max 16384 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 23, 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 2, 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "697ea9f7": { 2024-04-23T18:17:34.363 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 2, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 74.5, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:34.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:34.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:34.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:34.365 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 msgr2=0x7f79e40a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 0x7f79e40a6320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79fc1045c0 tx=0x7f79ec005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 msgr2=0x7f79fc102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79f8005940 tx=0x7f79f8004270 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 shutdown_connections 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79e40a3e60 0x7f79e40a6320 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc106170 0x7f79fc103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79fc1035b0 0x7f79fc1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 --2- 172.21.15.26:0/2485360695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79fc1010a0 0x7f79fc102b30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 >> 172.21.15.26:0/2485360695 conn(0x7f79fc077f50 msgr2=0x7f79fc069520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 shutdown_connections 2024-04-23T18:17:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:34.369+0000 7f7a02a8a640 1 -- 172.21.15.26:0/2485360695 wait complete. 2024-04-23T18:17:34.374 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '697ea9f7', quiesce took 0.6 seconds. Will release after: 14.5 2024-04-23T18:17:34.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:34 smithi110 ceph-mon[32585]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:34.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:34 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1721668845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:34.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:34 smithi026 ceph-mon[29364]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:34 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1721668845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:34 smithi131 ceph-mon[33041]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:34.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:34 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1721668845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:36.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:36 smithi110 ceph-mon[32585]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:36.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:36 smithi026 ceph-mon[29364]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:36.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:36 smithi131 ceph-mon[33041]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-23T18:17:37.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:37 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:37.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:37 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:37.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:37 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 -- 172.21.15.26:0/1906831971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b4107290 msgr2=0x7f92b410ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 --2- 172.21.15.26:0/1906831971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b4107290 0x7f92b410ee60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009ca0 tx=0x7f92ac02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 -- 172.21.15.26:0/1906831971 shutdown_connections 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 --2- 172.21.15.26:0/1906831971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4102b70 0x7f92b4106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 --2- 172.21.15.26:0/1906831971 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b4107290 0x7f92b410ee60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 --2- 172.21.15.26:0/1906831971 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b41021c0 0x7f92b41025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 -- 172.21.15.26:0/1906831971 >> 172.21.15.26:0/1906831971 conn(0x7f92b40fd9e0 msgr2=0x7f92b40ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 -- 172.21.15.26:0/1906831971 shutdown_connections 2024-04-23T18:17:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.150+0000 7f92ba937640 1 -- 172.21.15.26:0/1906831971 wait complete. 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.151+0000 7f92ba937640 1 Processor -- start 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.151+0000 7f92ba937640 1 -- start start 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4107290 0x7f92b4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b419ceb0 0x7f92b419d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41121e0 con 0x7f92b41021c0 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f92b4112060 con 0x7f92b4107290 2024-04-23T18:17:38.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92ba937640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f92b4112360 con 0x7f92b419ceb0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43446/0 (socket says 172.21.15.26:43446) 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 -- 172.21.15.26:0/3333816526 learned_addr learned my addr 172.21.15.26:0/3333816526 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b8ed4640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b419ceb0 0x7f92b419d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b37fe640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4107290 0x7f92b4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 -- 172.21.15.26:0/3333816526 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b419ceb0 msgr2=0x7f92b419d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b419ceb0 0x7f92b419d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 -- 172.21.15.26:0/3333816526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4107290 msgr2=0x7f92b4198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4107290 0x7f92b4198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.152+0000 7f92b3fff640 1 -- 172.21.15.26:0/3333816526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b419d9a0 con 0x7f92b41021c0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92a400c920 tx=0x7f92a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92b17fa640 1 -- 172.21.15.26:0/3333816526 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4007c70 con 0x7f92b41021c0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92b17fa640 1 -- 172.21.15.26:0/3333816526 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a4007e10 con 0x7f92b41021c0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92b17fa640 1 -- 172.21.15.26:0/3333816526 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4018820 con 0x7f92b41021c0 2024-04-23T18:17:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b419dc90 con 0x7f92b41021c0 2024-04-23T18:17:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.153+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b41a5430 con 0x7f92b41021c0 2024-04-23T18:17:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.155+0000 7f92b17fa640 1 -- 172.21.15.26:0/3333816526 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f92a400f450 con 0x7f92b41021c0 2024-04-23T18:17:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.156+0000 7f92b17fa640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 0x7f929c0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.156+0000 7f92b17fa640 1 -- 172.21.15.26:0/3333816526 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f92a40d0a50 con 0x7f92b41021c0 2024-04-23T18:17:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.156+0000 7f92b37fe640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 0x7f929c0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92b37fe640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 0x7f929c0a6420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92b4108840 tx=0x7f92a800b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 Processor -- start 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 -- start start 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d8282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d82abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e53d782970 con 0x55e53d81d620 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e53d71f4a0 con 0x55e53d828800 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92bd64d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e53d77e930 con 0x55e53d81df40 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43448/0 (socket says 172.21.15.26:43448) 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92b3fff640 1 -- 172.21.15.26:0/3945859222 learned_addr learned my addr 172.21.15.26:0/3945859222 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.157+0000 7f92b37fe640 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d8282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b8ed4640 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d82abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 -- 172.21.15.26:0/3945859222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 msgr2=0x55e53d8282c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d8282c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 -- 172.21.15.26:0/3945859222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 msgr2=0x55e53d82abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d82abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 -- 172.21.15.26:0/3945859222 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e53d495210 con 0x55e53d81d620 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f92b3fff640 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f92a4098cb0 tx=0x7f92a409d010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fa9da4e31d59eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f929a7fc640 1 -- 172.21.15.26:0/3945859222 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40a54c0 con 0x55e53d81d620 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f929a7fc640 1 -- 172.21.15.26:0/3945859222 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a400fa90 con 0x55e53d81d620 2024-04-23T18:17:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.158+0000 7f929a7fc640 1 -- 172.21.15.26:0/3945859222 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40a5290 con 0x55e53d81d620 2024-04-23T18:17:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 -- 172.21.15.26:0/3945859222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 msgr2=0x55e53d81da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f92a4098cb0 tx=0x7f92a409d010 comp rx=0 tx=0).stop 2024-04-23T18:17:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 -- 172.21.15.26:0/3945859222 shutdown_connections 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d82abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d8282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 --2- 172.21.15.26:0/3945859222 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81d620 0x55e53d81da00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 -- 172.21.15.26:0/3945859222 >> 172.21.15.26:0/3945859222 conn(0x55e53d819b30 msgr2=0x55e53d819f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 -- 172.21.15.26:0/3945859222 shutdown_connections 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.159+0000 7f92bd64d740 1 -- 172.21.15.26:0/3945859222 wait complete. 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 -- start start 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 Processor -- start 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81d620 0x55e53d828180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d8259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e53d82b430 con 0x55e53d81df40 2024-04-23T18:17:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e53d71d0b0 con 0x55e53d828800 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e53d82b5b0 con 0x55e53d81d620 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92bd64d740 10 client.0 _ll_register_callbacks cb 0x55e53d70ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92b37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92b37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43464/0 (socket says 172.21.15.26:43464) 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92b37fe640 1 -- 172.21.15.26:0/2570706800 learned_addr learned my addr 172.21.15.26:0/2570706800 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81d620 0x55e53d828180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.160+0000 7f92b8ed4640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d8259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 -- 172.21.15.26:0/2570706800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81d620 msgr2=0x55e53d828180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81d620 0x55e53d828180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 -- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 msgr2=0x55e53d8259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d8259f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 -- 172.21.15.26:0/2570706800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e53d561620 con 0x55e53d81df40 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f92a80116f0 tx=0x7f92a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea75883beabe5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92937fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8005220 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f92a80053c0 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a400fa90 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f92a80055c0 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92bd64d740 10 client.15612 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.161+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e53d83b7f0 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a400fa90 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f92a801fb30 con 0x55e53d81df40 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 10 client.15612 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 20 client.15612 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d83f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 4 client.15612 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e53d790390 con 0x55e53d83cd00 2024-04-23T18:17:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.162+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d83f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.163+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d83f0f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef325ccfea187b7 server_cookie=8adde67a31176740 in_seq=0 out_seq=0 2024-04-23T18:17:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.163+0000 7f92937fe640 10 client.15612 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.163+0000 7f92937fe640 1 -- 172.21.15.26:0/2570706800 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e53d790390 con 0x55e53d83cd00 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92937fe640 10 client.15612 handle_command_reply: tid=0 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 client.15612 shutdown 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 msgr2=0x55e53d825470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f92a80116f0 tx=0x7f92a8004730 comp rx=0 tx=0).stop 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 shutdown_connections 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d83f0f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d828800 0x55e53d8259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92937fe640 0 client.15612 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d825470 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2570706800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81d620 0x55e53d828180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 >> 172.21.15.26:0/2570706800 conn(0x55e53d819b30 msgr2=0x55e53d81f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 shutdown_connections 2024-04-23T18:17:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 1 -- 172.21.15.26:0/2570706800 wait complete. 2024-04-23T18:17:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.164+0000 7f92bd64d740 20 client.15612 trim_cache size 0 max 16384 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 Processor -- start 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 -- start start 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d8342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d827cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41ae1c0 con 0x55e53d828800 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41aba90 con 0x55e53d81d620 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92bd64d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41a98e0 con 0x55e53d81df40 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92b8ed4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92b8ed4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43476/0 (socket says 172.21.15.26:43476) 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.179+0000 7f92b8ed4640 1 -- 172.21.15.26:0/1178717296 learned_addr learned my addr 172.21.15.26:0/1178717296 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d8342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.174 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b37fe640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d827cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 -- 172.21.15.26:0/1178717296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 msgr2=0x55e53d827cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d827cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 -- 172.21.15.26:0/1178717296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 msgr2=0x55e53d844a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 -- 172.21.15.26:0/1178717296 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e53d825d20 con 0x55e53d81d620 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b8ed4640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f92b3fff640 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d8342f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e53d81e4b0 tx=0x7f92a400a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b920742d200cac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f9299ffb640 1 -- 172.21.15.26:0/1178717296 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4098ec0 con 0x55e53d81d620 2024-04-23T18:17:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f9299ffb640 1 -- 172.21.15.26:0/1178717296 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a40a5030 con 0x55e53d81d620 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 -- 172.21.15.26:0/1178717296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 msgr2=0x55e53d8342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d8342f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e53d81e4b0 tx=0x7f92a400a560 comp rx=0 tx=0).stop 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.180+0000 7f9299ffb640 1 -- 172.21.15.26:0/1178717296 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a409db80 con 0x55e53d81d620 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 -- 172.21.15.26:0/1178717296 shutdown_connections 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d828800 0x55e53d844a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d81df40 0x55e53d827cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 --2- 172.21.15.26:0/1178717296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d8342f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 -- 172.21.15.26:0/1178717296 >> 172.21.15.26:0/1178717296 conn(0x55e53d819b30 msgr2=0x55e53d820af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 -- 172.21.15.26:0/1178717296 shutdown_connections 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.181+0000 7f92bd64d740 1 -- 172.21.15.26:0/1178717296 wait complete. 2024-04-23T18:17:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 -- start start 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 Processor -- start 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d82c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d828800 0x55e53d82ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41a9f50 con 0x55e53d81df40 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41ab3c0 con 0x55e53d81d620 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92bd64d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41aa620 con 0x55e53d828800 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.182+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d82c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:57168/0 (socket says 172.21.15.26:57168) 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 -- 172.21.15.26:0/2000812880 learned_addr learned my addr 172.21.15.26:0/2000812880 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92bd64d740 10 client.0 _ll_register_callbacks cb 0x7f92a8010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 -- 172.21.15.26:0/2000812880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d828800 msgr2=0x55e53d82ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d828800 0x55e53d82ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 -- 172.21.15.26:0/2000812880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 msgr2=0x55e53d82c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d82c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 -- 172.21.15.26:0/2000812880 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e53d8a2470 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d82c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92b3fff640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92a4098c60 tx=0x7f92a40984a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63499c995eaa12 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92937fe640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.183+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4098ec0 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a40a5030 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f92a40ce360 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f92a4099060 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 10 client.25166 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e53d8a3720 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a40a5030 con 0x55e53d81d620 2024-04-23T18:17:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f92a40b2050 con 0x55e53d81d620 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 10 client.25166 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 10 client.25166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 20 client.25166 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d8a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 4 client.25166 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e53d78a200 con 0x55e53d83cd00 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.184+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d8a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.185+0000 7f92937fe640 10 client.25166 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.185+0000 7f92b37fe640 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d8a59b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae9a6eef7fbdb2 server_cookie=4998e40302e0c3f5 in_seq=0 out_seq=0 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92937fe640 1 -- 172.21.15.26:0/2000812880 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e53d78a200 con 0x55e53d83cd00 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92937fe640 10 client.25166 handle_command_reply: tid=0 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 client.25166 shutdown 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 msgr2=0x55e53d82c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92a4098c60 tx=0x7f92a40984a0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 shutdown_connections 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e53d83cd00 0x55e53d8a59b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e53d828800 0x55e53d82ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92937fe640 0 client.25166 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e53d81df40 0x55e53d82c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 --2- 172.21.15.26:0/2000812880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e53d81d620 0x55e53d82c3d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 >> 172.21.15.26:0/2000812880 conn(0x55e53d819b30 msgr2=0x55e53d8332e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 shutdown_connections 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.189+0000 7f92bd64d740 1 -- 172.21.15.26:0/2000812880 wait complete. 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.190+0000 7f92bd64d740 20 client.25166 trim_cache size 0 max 16384 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:17:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.190+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 msgr2=0x7f929c0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.190+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 0x7f929c0a6420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f92b4108840 tx=0x7f92a800b390 comp rx=0 tx=0).stop 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.190+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 msgr2=0x7f92b4198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.190+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92a400c920 tx=0x7f92a400cde0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 shutdown_connections 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f929c0a3f60 0x7f929c0a6420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b4107290 0x7f92b4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b419ceb0 0x7f92b419d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 --2- 172.21.15.26:0/3333816526 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f92b41021c0 0x7f92b4198640 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 >> 172.21.15.26:0/3333816526 conn(0x7f92b40fd9e0 msgr2=0x7f92b40fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 shutdown_connections 2024-04-23T18:17:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:38.191+0000 7f92ba937640 1 -- 172.21.15.26:0/3333816526 wait complete. 2024-04-23T18:17:38.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:38 smithi110 ceph-mon[32585]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-04-23T18:17:38.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:38 smithi026 ceph-mon[29364]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-04-23T18:17:38.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:38 smithi131 ceph-mon[33041]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-04-23T18:17:40.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:40 smithi110 ceph-mon[32585]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T18:17:40.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:40 smithi026 ceph-mon[29364]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T18:17:40.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:40 smithi131 ceph-mon[33041]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T18:17:42.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:42 smithi110 ceph-mon[32585]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-23T18:17:42.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:42 smithi026 ceph-mon[29364]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-23T18:17:42.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:42 smithi131 ceph-mon[33041]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-23T18:17:44.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:44 smithi110 ceph-mon[32585]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:44.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:44 smithi026 ceph-mon[29364]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:44.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:44 smithi131 ceph-mon[33041]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:46.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:46 smithi110 ceph-mon[32585]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:46.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:46 smithi026 ceph-mon[29364]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:46.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:46 smithi131 ceph-mon[33041]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-23T18:17:48.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:48.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:48 smithi026 ceph-mon[29364]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:17:48.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:48 smithi110 ceph-mon[32585]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 -- 172.21.15.26:0/1543249812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c103b00 msgr2=0x7f351c10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 --2- 172.21.15.26:0/1543249812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c103b00 0x7f351c10bfe0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3510009900 tx=0x7f351002a510 comp rx=0 tx=0).stop 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 -- 172.21.15.26:0/1543249812 shutdown_connections 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 --2- 172.21.15.26:0/1543249812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c10c520 0x7f351c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 --2- 172.21.15.26:0/1543249812 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c103b00 0x7f351c10bfe0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 --2- 172.21.15.26:0/1543249812 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c1031e0 0x7f351c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 -- 172.21.15.26:0/1543249812 >> 172.21.15.26:0/1543249812 conn(0x7f351c077f50 msgr2=0x7f351c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:48.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 -- 172.21.15.26:0/1543249812 shutdown_connections 2024-04-23T18:17:48.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.990+0000 7f352190f640 1 -- 172.21.15.26:0/1543249812 wait complete. 2024-04-23T18:17:48.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 Processor -- start 2024-04-23T18:17:48.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 -- start start 2024-04-23T18:17:48.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c1031e0 0x7f351c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:48.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c103b00 0x7f351c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f351c112200 con 0x7f351c10c520 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f351c112080 con 0x7f351c1031e0 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f352190f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f351c112380 con 0x7f351c103b00 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f351b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52040/0 (socket says 172.21.15.26:52040) 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f351b7fe640 1 -- 172.21.15.26:0/1673933642 learned_addr learned my addr 172.21.15.26:0/1673933642 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f351affd640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c1031e0 0x7f351c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.991+0000 7f351a7fc640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c103b00 0x7f351c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 -- 172.21.15.26:0/1673933642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c103b00 msgr2=0x7f351c19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c103b00 0x7f351c19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 -- 172.21.15.26:0/1673933642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c1031e0 msgr2=0x7f351c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c1031e0 0x7f351c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 -- 172.21.15.26:0/1673933642 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351c198ef0 con 0x7f351c10c520 2024-04-23T18:17:48.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f351b7fe640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f351400eb10 tx=0x7f351400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514004b20 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351c1991e0 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c1a54f0 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3514004cc0 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.992+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514010690 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.993+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3508005290 con 0x7f351c10c520 2024-04-23T18:17:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.995+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f35140108f0 con 0x7f351c10c520 2024-04-23T18:17:48.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.995+0000 7f352090d640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 0x7f35040a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:48.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.996+0000 7f351affd640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 0x7f35040a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:48.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.996+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f35140d0f90 con 0x7f351c10c520 2024-04-23T18:17:48.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:48.996+0000 7f351affd640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 0x7f35040a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f350c009b60 tx=0x7f350c005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:48.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:48 smithi131 ceph-mon[33041]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-23T18:17:48.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.000+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3514053ff0 con 0x7f351c10c520 2024-04-23T18:17:49.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.241+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3508005580 con 0x7f351c10c520 2024-04-23T18:17:49.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.241+0000 7f352090d640 1 -- 172.21.15.26:0/1673933642 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7f351409d4f0 con 0x7f351c10c520 2024-04-23T18:17:49.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:17:49.235 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:35.666815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:17:49.235 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2024-04-23T18:17:49.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 msgr2=0x7f35040a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 0x7f35040a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f350c009b60 tx=0x7f350c005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:49.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 msgr2=0x7f351c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f351400eb10 tx=0x7f351400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 shutdown_connections 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35040a3e30 0x7f35040a62f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f351c10c520 0x7f351c198630 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f351c103b00 0x7f351c19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 --2- 172.21.15.26:0/1673933642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f351c1031e0 0x7f351c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 >> 172.21.15.26:0/1673933642 conn(0x7f351c077f50 msgr2=0x7f351c0ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 shutdown_connections 2024-04-23T18:17:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.244+0000 7f352190f640 1 -- 172.21.15.26:0/1673933642 wait complete. 2024-04-23T18:17:49.249 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 697ea9f7 --release --await' 2024-04-23T18:17:49.249 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 697ea9f7 --release --await 2024-04-23T18:17:49.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 -- 172.21.15.26:0/2979377271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 msgr2=0x7f0434101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 --2- 172.21.15.26:0/2979377271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f0434101130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0424009860 tx=0x7f042402a340 comp rx=0 tx=0).stop 2024-04-23T18:17:49.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 -- 172.21.15.26:0/2979377271 shutdown_connections 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 --2- 172.21.15.26:0/2979377271 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 0x7f0434110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 --2- 172.21.15.26:0/2979377271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 0x7f043410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 --2- 172.21.15.26:0/2979377271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f0434101130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 -- 172.21.15.26:0/2979377271 >> 172.21.15.26:0/2979377271 conn(0x7f0434077f50 msgr2=0x7f0434078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 -- 172.21.15.26:0/2979377271 shutdown_connections 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.409+0000 7f0439b4b640 1 -- 172.21.15.26:0/2979377271 wait complete. 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 Processor -- start 2024-04-23T18:17:49.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 -- start start 2024-04-23T18:17:49.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 0x7f04341a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 0x7f043419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f04341143c0 con 0x7f043410e900 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0434114240 con 0x7f0434100d50 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f0439b4b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0434114540 con 0x7f0434101670 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.410+0000 7f04337fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f0433fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 0x7f043419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:47840/0 (socket says 172.21.15.26:47840) 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 -- 172.21.15.26:0/512090261 learned_addr learned my addr 172.21.15.26:0/512090261 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f0432ffd640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 0x7f04341a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 -- 172.21.15.26:0/512090261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 msgr2=0x7f04341a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 0x7f04341a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 -- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 msgr2=0x7f043419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 0x7f043419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 -- 172.21.15.26:0/512090261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043419b020 con 0x7f0434100d50 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f04337fe640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f04240303f0 tx=0x7f0424032eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f0430ff9640 1 -- 172.21.15.26:0/512090261 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424040020 con 0x7f0434100d50 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.411+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043419b2b0 con 0x7f0434100d50 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.412+0000 7f0430ff9640 1 -- 172.21.15.26:0/512090261 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0424004040 con 0x7f0434100d50 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.412+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04341a7620 con 0x7f0434100d50 2024-04-23T18:17:49.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.412+0000 7f0430ff9640 1 -- 172.21.15.26:0/512090261 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424037040 con 0x7f0434100d50 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f0430ff9640 1 -- 172.21.15.26:0/512090261 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f0424043020 con 0x7f0434100d50 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 Processor -- start 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 -- start start 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679afc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b06f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55c679a616d0 con 0x55c679b07480 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6799fe1a0 con 0x55c679afcdb0 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.414+0000 7f043c862740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55c679a5d690 con 0x55c679afc490 2024-04-23T18:17:49.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52060/0 (socket says 172.21.15.26:52060) 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 -- 172.21.15.26:0/1979350590 learned_addr learned my addr 172.21.15.26:0/1979350590 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0433fff640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679afc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f04337fe640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b06f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 -- 172.21.15.26:0/1979350590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 msgr2=0x55c679afc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679afc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 -- 172.21.15.26:0/1979350590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 msgr2=0x55c679b06f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b06f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 -- 172.21.15.26:0/1979350590 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c679a085b0 con 0x55c679b07480 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f04337fe640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b06f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0430ff9640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 0x7f041c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f042800bcb0 tx=0x7f04280107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db4355d4d68b2d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.415+0000 7f0430ff9640 1 -- 172.21.15.26:0/512090261 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f04240f01e0 con 0x7f0434100d50 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f0433fff640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 0x7f041c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f0419ffb640 1 -- 172.21.15.26:0/1979350590 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428004030 con 0x55c679b07480 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f0419ffb640 1 -- 172.21.15.26:0/1979350590 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04280041d0 con 0x55c679b07480 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f0419ffb640 1 -- 172.21.15.26:0/1979350590 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04280117b0 con 0x55c679b07480 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 -- 172.21.15.26:0/1979350590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 msgr2=0x55c679b09870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f042800bcb0 tx=0x7f04280107a0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f0433fff640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 0x7f041c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f043406b6e0 tx=0x7f042c010040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 -- 172.21.15.26:0/1979350590 shutdown_connections 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b06f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b09870 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 --2- 172.21.15.26:0/1979350590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679afc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 -- 172.21.15.26:0/1979350590 >> 172.21.15.26:0/1979350590 conn(0x55c679af87b0 msgr2=0x55c679af8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 -- 172.21.15.26:0/1979350590 shutdown_connections 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.416+0000 7f043c862740 1 -- 172.21.15.26:0/1979350590 wait complete. 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 -- start start 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 Processor -- start 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afcdb0 0x55c679b041e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679b07480 0x55c679b04920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55c679b0a310 con 0x55c679afc490 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6799fbb20 con 0x55c679b07480 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55c679b0a490 con 0x55c679afcdb0 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f043c862740 10 client.0 _ll_register_callbacks cb 0x55c6799fb530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:49.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f04337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.417+0000 7f04337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52064/0 (socket says 172.21.15.26:52064) 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 -- 172.21.15.26:0/1536319700 learned_addr learned my addr 172.21.15.26:0/1536319700 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afcdb0 0x55c679b041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0433fff640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679b07480 0x55c679b04920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 -- 172.21.15.26:0/1536319700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afcdb0 msgr2=0x55c679b041e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afcdb0 0x55c679b041e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 -- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679b07480 msgr2=0x55c679b04920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679b07480 0x55c679b04920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 -- 172.21.15.26:0/1536319700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c679773920 con 0x55c679afc490 2024-04-23T18:17:49.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f04337fe640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f04240bbe50 tx=0x7f04240bbfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d588f002d0c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04240f0020 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f04240edb50 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04280041d0 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.418+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f04240edd50 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 10 client.15636 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c67983fd30 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04280041d0 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f04240c4930 con 0x55c679afc490 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 10 client.15636 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 10 client.15636 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 20 client.15636 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:49.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b17670 0x55c679b19ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 4 client.15636 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:49.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c679a1f500 con 0x55c679b17670 2024-04-23T18:17:49.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.419+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b17670 0x55c679b19ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.420+0000 7f0412ffd640 10 client.15636 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:49.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.420+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b17670 0x55c679b19ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f81a3ce524e63f0 server_cookie=b2fbeb3426305dd0 in_seq=0 out_seq=0 2024-04-23T18:17:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f0412ffd640 1 -- 172.21.15.26:0/1536319700 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c679a1f500 con 0x55c679b17670 2024-04-23T18:17:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f0412ffd640 10 client.15636 handle_command_reply: tid=0 2024-04-23T18:17:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 client.15636 shutdown 2024-04-23T18:17:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 msgr2=0x55c679b06ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f04240bbe50 tx=0x7f04240bbfe0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 shutdown_connections 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b17670 0x55c679b19ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679b07480 0x55c679b04920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afcdb0 0x55c679b041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 --2- 172.21.15.26:0/1536319700 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b06ef0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 >> 172.21.15.26:0/1536319700 conn(0x55c679af87b0 msgr2=0x55c679afe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f0412ffd640 0 client.15636 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 shutdown_connections 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 1 -- 172.21.15.26:0/1536319700 wait complete. 2024-04-23T18:17:49.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.421+0000 7f043c862740 20 client.15636 trim_cache size 0 max 16384 2024-04-23T18:17:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.456+0000 7f043c862740 1 Processor -- start 2024-04-23T18:17:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.456+0000 7f043c862740 1 -- start start 2024-04-23T18:17:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679aff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b03160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f04341b24f0 con 0x55c679b07480 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04341afdc0 con 0x55c679afcdb0 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f043c862740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f04341adc10 con 0x55c679afc490 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52070/0 (socket says 172.21.15.26:52070) 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 -- 172.21.15.26:0/1243386497 learned_addr learned my addr 172.21.15.26:0/1243386497 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f04337fe640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679aff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0432ffd640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b03160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 -- 172.21.15.26:0/1243386497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 msgr2=0x55c679aff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679aff6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 -- 172.21.15.26:0/1243386497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 msgr2=0x55c679b03160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b03160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 -- 172.21.15.26:0/1243386497 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c679afbaa0 con 0x55c679b07480 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f04337fe640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679aff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.457+0000 7f0433fff640 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f042c003fd0 tx=0x7f042c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707a1fb58188c695 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.458+0000 7f04197fa640 1 -- 172.21.15.26:0/1243386497 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c00da30 con 0x55c679b07480 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.458+0000 7f04197fa640 1 -- 172.21.15.26:0/1243386497 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f042c02b070 con 0x55c679b07480 2024-04-23T18:17:49.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.458+0000 7f04197fa640 1 -- 172.21.15.26:0/1243386497 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c01a450 con 0x55c679b07480 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 -- 172.21.15.26:0/1243386497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 msgr2=0x55c679b06a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f042c003fd0 tx=0x7f042c016000 comp rx=0 tx=0).stop 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 -- 172.21.15.26:0/1243386497 shutdown_connections 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b03160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679b07480 0x55c679b06a20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 --2- 172.21.15.26:0/1243386497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679afc490 0x55c679aff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 -- 172.21.15.26:0/1243386497 >> 172.21.15.26:0/1243386497 conn(0x55c679af87b0 msgr2=0x55c679af79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 -- 172.21.15.26:0/1243386497 shutdown_connections 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.459+0000 7f043c862740 1 -- 172.21.15.26:0/1243386497 wait complete. 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.460+0000 7f043c862740 1 -- start start 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.460+0000 7f043c862740 1 Processor -- start 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.460+0000 7f043c862740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.460+0000 7f043c862740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b0fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f043c862740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679b07480 0x55c679b00490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52076/0 (socket says 172.21.15.26:52076) 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 -- 172.21.15.26:0/2507032702 learned_addr learned my addr 172.21.15.26:0/2507032702 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f04341ae280 con 0x55c679afc490 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04341af6f0 con 0x55c679afcdb0 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f04341ae950 con 0x55c679b07480 2024-04-23T18:17:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f0432ffd640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b0fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f043c862740 10 client.0 _ll_register_callbacks cb 0x7f04240b83f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f0433fff640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679b07480 0x55c679b00490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 -- 172.21.15.26:0/2507032702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679b07480 msgr2=0x55c679b00490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679b07480 0x55c679b00490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 -- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 msgr2=0x55c679b0fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b0fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.461+0000 7f04337fe640 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c679b00b00 con 0x55c679afc490 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f04337fe640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04240b85b0 tx=0x7f04240bc050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1786021c632677 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f0412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04240bbab0 con 0x55c679afc490 2024-04-23T18:17:49.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f04240bb7b0 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f042c02b070 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f04240bc150 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f043c862740 10 client.15642 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.462+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c679b01e20 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f042c02b070 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f04240c1690 con 0x55c679afc490 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 10 client.15642 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 10 client.15642 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 20 client.15642 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b7ec70 0x55c679b81060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 4 client.15642 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "697ea9f7", "release": true, "await": true}] 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "697ea9f7", "release": true, "await": true}) -- 0x55c679a6f0f0 con 0x55c679b7ec70 2024-04-23T18:17:49.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.463+0000 7f0432ffd640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b7ec70 0x55c679b81060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.464+0000 7f0412ffd640 10 client.15642 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:49.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.464+0000 7f0432ffd640 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b7ec70 0x55c679b81060 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a8cf8b6d031c0b server_cookie=25d9c8be62adeb0b in_seq=0 out_seq=0 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f0412ffd640 1 -- 172.21.15.26:0/2507032702 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55c679a6f0f0 con 0x55c679b7ec70 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f0412ffd640 10 client.15642 handle_command_reply: tid=0 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 client.15642 shutdown 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 msgr2=0x55c679b0f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04240b85b0 tx=0x7f04240bc050 comp rx=0 tx=0).stop 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 shutdown_connections 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c679b7ec70 0x55c679b81060 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.465+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c679afcdb0 0x55c679b0fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c679b07480 0x55c679b00490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0412ffd640 0 client.15642 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 1 --2- 172.21.15.26:0/2507032702 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c679afc490 0x55c679b0f710 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 >> 172.21.15.26:0/2507032702 conn(0x55c679af87b0 msgr2=0x55c679afdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 shutdown_connections 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 1 -- 172.21.15.26:0/2507032702 wait complete. 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f043c862740 20 client.15642 trim_cache size 0 max 16384 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 24, 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 4, 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:17:49.463 INFO:teuthology.orchestra.run.smithi026.stdout: "697ea9f7": { 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 4, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 74.5, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:17:49.464 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:17:49.465 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 msgr2=0x7f041c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 0x7f041c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f043406b6e0 tx=0x7f042c010040 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 msgr2=0x7f04341a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f04240303f0 tx=0x7f0424032eb0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 shutdown_connections 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f041c0a3db0 0x7f041c0a6270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f043410e900 0x7f043419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0434101670 0x7f04341a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 --2- 172.21.15.26:0/512090261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0434100d50 0x7f04341a06a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.466+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 >> 172.21.15.26:0/512090261 conn(0x7f0434077f50 msgr2=0x7f04341100a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.467+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 shutdown_connections 2024-04-23T18:17:49.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:17:49.467+0000 7f0439b4b640 1 -- 172.21.15.26:0/512090261 wait complete. 2024-04-23T18:17:49.475 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '697ea9f7', seconds elapsed: 15.7 2024-04-23T18:17:49.586 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:49 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1673933642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:49.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:49 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1673933642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:49.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:49 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1673933642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:50.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:50 smithi110 ceph-mon[32585]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:17:50.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:50 smithi026 ceph-mon[29364]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:17:50.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:50 smithi131 ceph-mon[33041]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 -- 172.21.15.26:0/3525309616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 msgr2=0x7f3ab010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 --2- 172.21.15.26:0/3525309616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab010dd40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009ca0 tx=0x7f3aa802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 -- 172.21.15.26:0/3525309616 shutdown_connections 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 --2- 172.21.15.26:0/3525309616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab0101aa0 0x7f3ab0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 --2- 172.21.15.26:0/3525309616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab010dd40 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 --2- 172.21.15.26:0/3525309616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 -- 172.21.15.26:0/3525309616 >> 172.21.15.26:0/3525309616 conn(0x7f3ab0077fd0 msgr2=0x7f3ab00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 -- 172.21.15.26:0/3525309616 shutdown_connections 2024-04-23T18:17:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.331+0000 7f3ab5223640 1 -- 172.21.15.26:0/3525309616 wait complete. 2024-04-23T18:17:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.333+0000 7f3ab5223640 1 Processor -- start 2024-04-23T18:17:51.326 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.333+0000 7f3ab5223640 1 -- start start 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.333+0000 7f3ab5223640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.333+0000 7f3ab5223640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3ab5223640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 0x7f3ab019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3ab5223640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0110010 con 0x7f3ab0106170 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3ab5223640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab010fe90 con 0x7f3ab019ad00 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aa7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52090/0 (socket says 172.21.15.26:52090) 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aa7fff640 1 -- 172.21.15.26:0/2603840202 learned_addr learned my addr 172.21.15.26:0/2603840202 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0110190 con 0x7f3ab01010f0 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 0x7f3ab019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 -- 172.21.15.26:0/2603840202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 msgr2=0x7f3ab019b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 0x7f3ab019b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 -- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 msgr2=0x7f3ab01969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaed9d640 1 -- 172.21.15.26:0/2603840202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab019b7f0 con 0x7f3ab01010f0 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.334+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 0x7f3ab019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab0196490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009740 tx=0x7f3aa0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.330 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3aacd99640 1 -- 172.21.15.26:0/2603840202 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0009e50 con 0x7f3ab01010f0 2024-04-23T18:17:51.330 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab019bae0 con 0x7f3ab01010f0 2024-04-23T18:17:51.330 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3aacd99640 1 -- 172.21.15.26:0/2603840202 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa0005020 con 0x7f3ab01010f0 2024-04-23T18:17:51.330 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3aacd99640 1 -- 172.21.15.26:0/2603840202 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0004c00 con 0x7f3ab01010f0 2024-04-23T18:17:51.330 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.335+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab01a31c0 con 0x7f3ab01010f0 2024-04-23T18:17:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3aacd99640 1 -- 172.21.15.26:0/2603840202 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3aa001d020 con 0x7f3ab01010f0 2024-04-23T18:17:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 Processor -- start 2024-04-23T18:17:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 -- start start 2024-04-23T18:17:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df0612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 0x5653df063bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5653defbb970 con 0x5653df056620 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5653def584a0 con 0x5653df056f40 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3ab8011740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5653defb7930 con 0x5653df061800 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.339+0000 7f3aaf59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aaf59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52100/0 (socket says 172.21.15.26:52100) 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aaf59e640 1 -- 172.21.15.26:0/2678076844 learned_addr learned my addr 172.21.15.26:0/2678076844 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 0x5653df063bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df0612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 -- 172.21.15.26:0/2678076844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 msgr2=0x5653df0612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df0612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 -- 172.21.15.26:0/2678076844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 msgr2=0x5653df056a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 -- 172.21.15.26:0/2678076844 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653decce210 con 0x5653df061800 2024-04-23T18:17:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 0x5653df063bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ab01076f0 tx=0x7f3a98009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a60324899adc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aacd99640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 0x7f3a9c0a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.340+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 0x7f3a9c0a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.341+0000 7f3aa57fa640 1 -- 172.21.15.26:0/2678076844 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98009e50 con 0x5653df061800 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.341+0000 7f3aacd99640 1 -- 172.21.15.26:0/2603840202 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3aa00cfe00 con 0x7f3ab01010f0 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.341+0000 7f3aa57fa640 1 -- 172.21.15.26:0/2678076844 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9800d8c0 con 0x5653df061800 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.341+0000 7f3aa57fa640 1 -- 172.21.15.26:0/2678076844 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98005280 con 0x5653df061800 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- 172.21.15.26:0/2678076844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 msgr2=0x5653df063bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 0x7f3a9c0a6540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009a60 tx=0x7f3aa8005d80 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 0x5653df063bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ab01076f0 tx=0x7f3a98009b20 comp rx=0 tx=0).stop 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- 172.21.15.26:0/2678076844 shutdown_connections 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df0612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df061800 0x5653df063bf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2678076844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df056a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- 172.21.15.26:0/2678076844 >> 172.21.15.26:0/2678076844 conn(0x5653df052b30 msgr2=0x5653df052f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- 172.21.15.26:0/2678076844 shutdown_connections 2024-04-23T18:17:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- 172.21.15.26:0/2678076844 wait complete. 2024-04-23T18:17:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 -- start start 2024-04-23T18:17:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.342+0000 7f3ab8011740 1 Processor -- start 2024-04-23T18:17:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056620 0x5653df0610e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df05e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5653df06cc40 con 0x5653df061800 2024-04-23T18:17:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5653def560b0 con 0x5653df056f40 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5653df06cdc0 con 0x5653df056620 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52116/0 (socket says 172.21.15.26:52116) 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 -- 172.21.15.26:0/2806464292 learned_addr learned my addr 172.21.15.26:0/2806464292 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3ab8011740 10 client.0 _ll_register_callbacks cb 0x5653def43d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056620 0x5653df0610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df05e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 -- 172.21.15.26:0/2806464292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056620 msgr2=0x5653df0610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056620 0x5653df0610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 -- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 msgr2=0x5653df05e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df05e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aaf59e640 1 -- 172.21.15.26:0/2806464292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653ded9a620 con 0x5653df061800 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.343+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df05e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80319d0 tx=0x7f3aa802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f253f241042786 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3a967fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8004250 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3aa80043f0 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9800d8c0 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3aa80045f0 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3ab8011740 10 client.15654 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.344+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5653df078ae0 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa8036040 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3aa8052070 con 0x5653df061800 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 10 client.15654 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 20 client.15654 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df07a010 0x5653df07c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 4 client.15654 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a9800d8c0 con 0x5653df07a010 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.345+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df07a010 0x5653df07c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3a967fc640 10 client.15654 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df07a010 0x5653df07c490 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b61c92154309583 server_cookie=e3d1e03f406ecb08 in_seq=0 out_seq=0 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3a967fc640 1 -- 172.21.15.26:0/2806464292 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3a9800d8c0 con 0x5653df07a010 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3a967fc640 10 client.15654 handle_command_reply: tid=0 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3ab8011740 1 client.15654 shutdown 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 msgr2=0x5653df05e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.346+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80319d0 tx=0x7f3aa802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 shutdown_connections 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df07a010 0x5653df07c490 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df056f40 0x5653df05e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df061800 0x5653df05e910 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 --2- 172.21.15.26:0/2806464292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056620 0x5653df0610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 >> 172.21.15.26:0/2806464292 conn(0x5653df052b30 msgr2=0x5653df0588b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3a967fc640 0 client.15654 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 shutdown_connections 2024-04-23T18:17:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 1 -- 172.21.15.26:0/2806464292 wait complete. 2024-04-23T18:17:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.347+0000 7f3ab8011740 20 client.15654 trim_cache size 0 max 16384 2024-04-23T18:17:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 Processor -- start 2024-04-23T18:17:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 -- start start 2024-04-23T18:17:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df081d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df0822a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01ae0b0 con 0x5653df056620 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01ab980 con 0x5653df061800 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3ab8011740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01a97f0 con 0x5653df056f40 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.377+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52118/0 (socket says 172.21.15.26:52118) 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 -- 172.21.15.26:0/63270443 learned_addr learned my addr 172.21.15.26:0/63270443 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df081d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df0822a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 -- 172.21.15.26:0/63270443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 msgr2=0x5653df081d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df081d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 -- 172.21.15.26:0/63270443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 msgr2=0x5653df0822a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df0822a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 -- 172.21.15.26:0/63270443 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653df057530 con 0x5653df056620 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009710 tx=0x7f3aa00d22c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3bd2b1321d1ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.378+0000 7f3aa4ff9640 1 -- 172.21.15.26:0/63270443 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa009ffb0 con 0x5653df056620 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.379+0000 7f3aa4ff9640 1 -- 172.21.15.26:0/63270443 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa00a4870 con 0x5653df056620 2024-04-23T18:17:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.379+0000 7f3aa4ff9640 1 -- 172.21.15.26:0/63270443 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0098270 con 0x5653df056620 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.379+0000 7f3ab8011740 1 -- 172.21.15.26:0/63270443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 msgr2=0x5653df060c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.379+0000 7f3ab8011740 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009710 tx=0x7f3aa00d22c0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 -- 172.21.15.26:0/63270443 shutdown_connections 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df0822a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df081d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 --2- 172.21.15.26:0/63270443 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df060c10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 -- 172.21.15.26:0/63270443 >> 172.21.15.26:0/63270443 conn(0x5653df052b30 msgr2=0x5653df05e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 -- 172.21.15.26:0/63270443 shutdown_connections 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.380+0000 7f3ab8011740 1 -- 172.21.15.26:0/63270443 wait complete. 2024-04-23T18:17:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3ab8011740 1 -- start start 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3ab8011740 1 Processor -- start 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3ab8011740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df0d8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3aaed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52126/0 (socket says 172.21.15.26:52126) 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.381+0000 7f3aaed9d640 1 -- 172.21.15.26:0/431601337 learned_addr learned my addr 172.21.15.26:0/431601337 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df06f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01a9e40 con 0x5653df056620 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01ab2b0 con 0x5653df061800 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01aa510 con 0x5653df056f40 2024-04-23T18:17:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3ab8011740 10 client.0 _ll_register_callbacks cb 0x7f3aa8031890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df0d8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 -- 172.21.15.26:0/431601337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 msgr2=0x5653df0d8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df0d8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df06f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 -- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 msgr2=0x5653df06f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df06f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653df06fa20 con 0x5653df056620 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaf59e640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3aaed9d640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3aa009bb90 tx=0x7f3aa00d0130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1620625b77b1994c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.382+0000 7f3a967fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00b5170 con 0x5653df056620 2024-04-23T18:17:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3aa00051d0 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa00a4870 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3aa0098270 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3ab8011740 10 client.15660 fetch_fsmap learned FSMap version 24 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5653df0e00e0 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa00bc0b0 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.383+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3aa00cd710 con 0x5653df056620 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 10 client.15660 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 10 client.15660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 20 client.15660 populate_metadata read hostname 'smithi026' 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df0e06e0 0x5653df0e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 4 client.15660 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3aa00a4870 con 0x5653df0e06e0 2024-04-23T18:17:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.384+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df0e06e0 0x5653df0e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.385+0000 7f3aa7fff640 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df0e06e0 0x5653df0e2b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965853fea8862a79 server_cookie=8cb0f1f3b6f7e6be in_seq=0 out_seq=0 2024-04-23T18:17:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.385+0000 7f3a967fc640 10 client.15660 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3a967fc640 1 -- 172.21.15.26:0/431601337 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3aa00a4870 con 0x5653df0e06e0 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3a967fc640 10 client.15660 handle_command_reply: tid=0 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 client.15660 shutdown 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 msgr2=0x5653df0d8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3aa009bb90 tx=0x7f3aa00d0130 comp rx=0 tx=0).stop 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 shutdown_connections 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5653df0e06e0 0x5653df0e2b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5653df061800 0x5653df06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5653df056f40 0x5653df0d8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 --2- 172.21.15.26:0/431601337 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5653df056620 0x5653df0d8950 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 >> 172.21.15.26:0/431601337 conn(0x5653df052b30 msgr2=0x5653df051420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3a967fc640 0 client.15660 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 shutdown_connections 2024-04-23T18:17:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 1 -- 172.21.15.26:0/431601337 wait complete. 2024-04-23T18:17:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.392+0000 7f3ab8011740 20 client.15660 trim_cache size 0 max 16384 2024-04-23T18:17:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:17:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:17:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:17:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 msgr2=0x7f3a9c0a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 0x7f3a9c0a6540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009a60 tx=0x7f3aa8005d80 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 msgr2=0x7f3ab0196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab0196490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009740 tx=0x7f3aa0009c00 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 shutdown_connections 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a9c0a4080 0x7f3a9c0a6540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab019ad00 0x7f3ab019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3ab0106170 0x7f3ab01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 --2- 172.21.15.26:0/2603840202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ab01010f0 0x7f3ab0196490 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 >> 172.21.15.26:0/2603840202 conn(0x7f3ab0077fd0 msgr2=0x7f3ab00feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 shutdown_connections 2024-04-23T18:17:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:17:51.393+0000 7f3ab5223640 1 -- 172.21.15.26:0/2603840202 wait complete. 2024-04-23T18:17:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:52 smithi110 ceph-mon[32585]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:17:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:52 smithi026 ceph-mon[29364]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:17:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:52 smithi131 ceph-mon[33041]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:17:54.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:54 smithi110 ceph-mon[32585]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:54.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:54 smithi026 ceph-mon[29364]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:54.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:54 smithi131 ceph-mon[33041]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:56.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:56 smithi110 ceph-mon[32585]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:17:56.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:56 smithi026 ceph-mon[29364]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:17:56.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:56 smithi131 ceph-mon[33041]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-23T18:17:58.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:17:58 smithi110 ceph-mon[32585]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:17:58.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:17:58 smithi026 ceph-mon[29364]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:17:58.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:17:58 smithi131 ceph-mon[33041]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:18:00.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:00 smithi110 ceph-mon[32585]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:00.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-mon[29364]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:00.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:00 smithi131 ceph-mon[33041]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:01.458 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:18:00.989+0000 7f7b07950700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:01.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:18:00.989+0000 7f7b07950700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:01.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:18:00.989+0000 7f7b07950700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:01.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:18:00.989+0000 7f7b07950700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:01.459 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:18:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:18:00.989+0000 7f7b07950700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:02.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:02 smithi110 ceph-mon[32585]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-23T18:18:02.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:02 smithi110 ceph-mon[32585]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-23T18:18:02.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:02 smithi026 ceph-mon[29364]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-23T18:18:02.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:02 smithi026 ceph-mon[29364]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-23T18:18:02.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:02 smithi131 ceph-mon[33041]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-23T18:18:02.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:02 smithi131 ceph-mon[33041]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-23T18:18:03.942 INFO:tasks.workunit.client.0.smithi026.stderr:Updating files: 97% (12173/12484) Updating files: 98% (12235/12484) Updating files: 99% (12360/12484) Updating files: 100% (12484/12484) Updating files: 100% (12484/12484), done. 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 -- 172.21.15.26:0/3244120994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e4410c520 msgr2=0x7f5e4410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/3244120994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e4410c520 0x7f5e4410e9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009ca0 tx=0x7f5e3c02a9e0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 -- 172.21.15.26:0/3244120994 shutdown_connections 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/3244120994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e44103b00 0x7f5e4410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/3244120994 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e4410c520 0x7f5e4410e9c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/3244120994 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 0x7f5e441035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 -- 172.21.15.26:0/3244120994 >> 172.21.15.26:0/3244120994 conn(0x7f5e44077fd0 msgr2=0x7f5e440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 -- 172.21.15.26:0/3244120994 shutdown_connections 2024-04-23T18:18:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.486+0000 7f5e4c69a640 1 -- 172.21.15.26:0/3244120994 wait complete. 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 Processor -- start 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 -- start start 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 0x7f5e44100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e4410c520 0x7f5e441018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5e44112190 con 0x7f5e44103b00 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e44112010 con 0x7f5e4410c520 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4c69a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5e44112310 con 0x7f5e441031e0 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e49c35640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e49c35640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59350/0 (socket says 172.21.15.26:59350) 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e49c35640 1 -- 172.21.15.26:0/1383463716 learned_addr learned my addr 172.21.15.26:0/1383463716 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e4410c520 0x7f5e441018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.487+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 0x7f5e44100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 -- 172.21.15.26:0/1383463716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 msgr2=0x7f5e44100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 0x7f5e44100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 -- 172.21.15.26:0/1383463716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e4410c520 msgr2=0x7f5e441018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e4410c520 0x7f5e441018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 -- 172.21.15.26:0/1383463716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e441a6b50 con 0x7f5e44103b00 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e49c35640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400d950 tx=0x7f5e3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e3b7fe640 1 -- 172.21.15.26:0/1383463716 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e34017070 con 0x7f5e44103b00 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e441a6e40 con 0x7f5e44103b00 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e3b7fe640 1 -- 172.21.15.26:0/1383463716 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e34009c30 con 0x7f5e44103b00 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e3b7fe640 1 -- 172.21.15.26:0/1383463716 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3400f620 con 0x7f5e44103b00 2024-04-23T18:18:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.488+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e441a7300 con 0x7f5e44103b00 2024-04-23T18:18:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.490+0000 7f5e4f3b5740 1 Processor -- start 2024-04-23T18:18:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 -- start start 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b5978722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b597874bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b5977cc970 con 0x55b597867620 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b5977694a0 con 0x55b597872800 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b5977c8930 con 0x55b597867f40 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59364/0 (socket says 172.21.15.26:59364) 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 -- 172.21.15.26:0/1549504695 learned_addr learned my addr 172.21.15.26:0/1549504695 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b5978722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e3b7fe640 1 -- 172.21.15.26:0/1383463716 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f5e34010460 con 0x7f5e44103b00 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 -- 172.21.15.26:0/1549504695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 msgr2=0x55b5978722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b5978722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 -- 172.21.15.26:0/1549504695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 msgr2=0x55b597874bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:18:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b597874bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 -- 172.21.15.26:0/1549504695 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5974df210 con 0x55b597867620 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.491+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c005fd0 tx=0x7f5e3c02eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f97384ac3ab60 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e38ff9640 1 -- 172.21.15.26:0/1549504695 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c049070 con 0x55b597867620 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e38ff9640 1 -- 172.21.15.26:0/1549504695 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e3c02e840 con 0x55b597867620 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e38ff9640 1 -- 172.21.15.26:0/1549504695 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c033040 con 0x55b597867620 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e3b7fe640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 0x7f5e300a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 0x7f5e300a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e3b7fe640 1 -- 172.21.15.26:0/1383463716 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5e340d0510 con 0x7f5e44103b00 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1549504695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 msgr2=0x55b597867a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c005fd0 tx=0x7f5e3c02eac0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1549504695 shutdown_connections 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.492+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b597874bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b5978722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1549504695 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b597867a00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1549504695 >> 172.21.15.26:0/1549504695 conn(0x55b597863b30 msgr2=0x55b597863f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 0x7f5e300a63d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55b597868bc0 tx=0x7f5e4000b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1549504695 shutdown_connections 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1549504695 wait complete. 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- start start 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 Processor -- start 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b59786f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59786f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b5978753d0 con 0x55b597867620 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b5977670b0 con 0x55b597872800 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b597875550 con 0x55b597867f40 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4f3b5740 10 client.0 _ll_register_callbacks cb 0x55b597754d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.493+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59374/0 (socket says 172.21.15.26:59374) 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 -- 172.21.15.26:0/3810838072 learned_addr learned my addr 172.21.15.26:0/3810838072 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e49c35640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b59786f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59786f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 -- 172.21.15.26:0/3810838072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 msgr2=0x55b59786f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b59786f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 -- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 msgr2=0x55b59786f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59786f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 -- 172.21.15.26:0/3810838072 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5975ab620 con 0x55b597867620 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59786f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4a436640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e40003e20 tx=0x7f5e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f39e66647bfb49 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4001b660 con 0x55b597867620 2024-04-23T18:18:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5e4001b800 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e3c02e840 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e4001ba00 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4f3b5740 10 client.15678 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.494+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5978812d0 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e3c02e840 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e40024070 con 0x55b597867620 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 10 client.15678 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 20 client.15678 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b597884bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 4 client.15678 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5977da390 con 0x55b5978827e0 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.495+0000 7f5e49c35640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b597884bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e2dffb640 10 client.15678 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e49c35640 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b597884bd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d20cff195e60605 server_cookie=e031f996e12de11b in_seq=0 out_seq=0 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e2dffb640 1 -- 172.21.15.26:0/3810838072 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5977da390 con 0x55b5978827e0 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e2dffb640 10 client.15678 handle_command_reply: tid=0 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e4f3b5740 1 client.15678 shutdown 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 msgr2=0x55b5978720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.496+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e40003e20 tx=0x7f5e400046c0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 shutdown_connections 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b597884bd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e2dffb640 0 client.15678 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59786f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b59786f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/3810838072 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b5978720d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 >> 172.21.15.26:0/3810838072 conn(0x55b597863b30 msgr2=0x55b597869860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 shutdown_connections 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/3810838072 wait complete. 2024-04-23T18:18:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.497+0000 7f5e4f3b5740 20 client.15678 trim_cache size 0 max 16384 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 Processor -- start 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 -- start start 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b597871c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59788a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441b2600 con 0x55b597867620 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441afed0 con 0x55b597872800 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441add20 con 0x55b597867f40 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.512+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59382/0 (socket says 172.21.15.26:59382) 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 -- 172.21.15.26:0/1964949028 learned_addr learned my addr 172.21.15.26:0/1964949028 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59788a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e49c35640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b597871c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 -- 172.21.15.26:0/1964949028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 msgr2=0x55b597871c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b597871c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 -- 172.21.15.26:0/1964949028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 msgr2=0x55b59788a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59788a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 -- 172.21.15.26:0/1964949028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b59786fc60 con 0x55b597867620 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59788a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e4a436640 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40012c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63f3c686a34ae05 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e2ffff640 1 -- 172.21.15.26:0/1964949028 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40033030 con 0x55b597867620 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e2ffff640 1 -- 172.21.15.26:0/1964949028 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4001bd60 con 0x55b597867620 2024-04-23T18:18:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.513+0000 7f5e2ffff640 1 -- 172.21.15.26:0/1964949028 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40004990 con 0x55b597867620 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1964949028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 msgr2=0x55b59787e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40012c70 comp rx=0 tx=0).stop 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1964949028 shutdown_connections 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597872800 0x55b59788a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867f40 0x55b597871c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/1964949028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597867620 0x55b59787e290 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1964949028 >> 172.21.15.26:0/1964949028 conn(0x55b597863b30 msgr2=0x55b597869860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1964949028 shutdown_connections 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/1964949028 wait complete. 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 -- start start 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.514+0000 7f5e4f3b5740 1 Processor -- start 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 0x55b597875e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 0x55b5978763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441ae390 con 0x55b597872800 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441af800 con 0x55b597867f40 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441aea60 con 0x55b597867620 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e49c35640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 0x55b597875e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4ac37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 0x55b5978763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:57942/0 (socket says 172.21.15.26:57942) 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e49c35640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 0x55b597875e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:39156/0 (socket says 172.21.15.26:39156) 2024-04-23T18:18:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 -- 172.21.15.26:0/138893955 learned_addr learned my addr 172.21.15.26:0/138893955 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4f3b5740 10 client.0 _ll_register_callbacks cb 0x55b597754d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 -- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 msgr2=0x55b597875e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 0x55b597875e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 -- 172.21.15.26:0/138893955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 msgr2=0x55b5978763a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 0x55b5978763a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4a436640 1 -- 172.21.15.26:0/138893955 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b597876ba0 con 0x55b597867620 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.515+0000 7f5e4ac37640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 0x55b5978763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e4a436640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e40005d80 tx=0x7f5e4001b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fdb0ba5cc49bf80 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4003c070 con 0x55b597867620 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4001bd60 con 0x55b597867620 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5e40026850 con 0x55b597867620 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e40026a90 con 0x55b597867620 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e4f3b5740 10 client.25207 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5978e8c40 con 0x55b597867620 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4001bd60 con 0x55b597867620 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.516+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e40024070 con 0x55b597867620 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 10 client.25207 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 10 client.25207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 20 client.25207 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b5978eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 4 client.25207 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5977d4200 con 0x55b5978827e0 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e49c35640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b5978eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e2dffb640 10 client.25207 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:04.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:04.517+0000 7f5e49c35640 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b5978eb1f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5502017099552c0 server_cookie=f5fc969cdc19aed7 in_seq=0 out_seq=0 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:Note: switching to '57e0caf76fdb7ab8b1358588f292d08519163844'. 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T18:18:04.642 INFO:tasks.workunit.client.0.smithi026.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: git switch -c 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr:Or undo this operation with: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: git switch - 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr: 2024-04-23T18:18:04.643 INFO:tasks.workunit.client.0.smithi026.stderr:HEAD is now at 57e0caf76fd qa/suites/fs: add quiescer to the fs suite 2024-04-23T18:18:04.644 DEBUG:teuthology.orchestra.run.smithi026:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T18:18:04.676 INFO:tasks.workunit.client.0.smithi026.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T18:18:04.678 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T18:18:04.678 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T18:18:04.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:04 smithi110 ceph-mon[32585]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:04.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:04 smithi026 ceph-mon[29364]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:04.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:04 smithi131 ceph-mon[33041]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-23T18:18:05.258 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T18:18:05.332 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T18:18:05.373 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T18:18:05.374 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T18:18:05.374 INFO:tasks.workunit.client.0.smithi026.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T18:18:05.413 INFO:tasks.workunit.client.0.smithi026.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T18:18:05.417 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:18:05.418 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T18:18:05.474 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-23T18:18:05.475 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-23T18:18:05.475 DEBUG:teuthology.orchestra.run.smithi026:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-23T18:18:05.538 INFO:tasks.workunit.client.0.smithi026.stderr:+ dbench 1 2024-04-23T18:18:05.555 INFO:tasks.workunit.client.0.smithi026.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T18:18:05.555 INFO:tasks.workunit.client.0.smithi026.stdout: 2024-04-23T18:18:05.555 INFO:tasks.workunit.client.0.smithi026.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T18:18:05.555 INFO:tasks.workunit.client.0.smithi026.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-23T18:18:05.656 INFO:tasks.workunit.client.0.smithi026.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-23T18:18:05.656 INFO:tasks.workunit.client.0.smithi026.stdout:releasing clients 2024-04-23T18:18:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.669+0000 7f5e2dffb640 1 -- 172.21.15.26:0/138893955 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5977d4200 con 0x55b5978827e0 2024-04-23T18:18:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.669+0000 7f5e2dffb640 10 client.25207 handle_command_reply: tid=0 2024-04-23T18:18:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.669+0000 7f5e4f3b5740 1 client.25207 shutdown 2024-04-23T18:18:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 msgr2=0x55b59787aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e40005d80 tx=0x7f5e4001b690 comp rx=0 tx=0).stop 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 shutdown_connections 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b5978827e0 0x55b5978eb1f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b597867f40 0x55b597875e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b597872800 0x55b5978763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 --2- 172.21.15.26:0/138893955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b597867620 0x55b59787aaf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e2dffb640 0 client.25207 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 >> 172.21.15.26:0/138893955 conn(0x55b597863b30 msgr2=0x55b597868fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 shutdown_connections 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 1 -- 172.21.15.26:0/138893955 wait complete. 2024-04-23T18:18:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4f3b5740 20 client.25207 trim_cache size 0 max 16384 2024-04-23T18:18:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:18:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:18:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:18:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 msgr2=0x7f5e300a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 0x7f5e300a63d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55b597868bc0 tx=0x7f5e4000b390 comp rx=0 tx=0).stop 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 msgr2=0x7f5e44101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400d950 tx=0x7f5e3400de10 comp rx=0 tx=0).stop 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 shutdown_connections 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5e300a3f10 0x7f5e300a63d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e4410c520 0x7f5e441018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5e44103b00 0x7f5e44101370 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 --2- 172.21.15.26:0/1383463716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e441031e0 0x7f5e44100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 >> 172.21.15.26:0/1383463716 conn(0x7f5e44077fd0 msgr2=0x7f5e440fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.670+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 shutdown_connections 2024-04-23T18:18:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:05.671+0000 7f5e4c69a640 1 -- 172.21.15.26:0/1383463716 wait complete. 2024-04-23T18:18:06.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 3197 34.28 MB/sec warmup 1 sec latency 7.249 ms 2024-04-23T18:18:06.741 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-mon[33041]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: *** Caught signal (Segmentation fault) ** 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: in thread 7ff497f77700 thread_name:md_log_replay 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4a9338d20] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2: /usr/bin/ceph-mds(+0x549372) [0x561a0e47c372] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561a0e47fb23] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 4: (MDLog::_replay_thread()+0x617) [0x561a0e4da647] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561a0e4e3d31] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff4a932e1ca] 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 7: clone() 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.475+0000 7ff497f77700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: in thread 7ff497f77700 thread_name:md_log_replay 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.743 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4a9338d20] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2: /usr/bin/ceph-mds(+0x549372) [0x561a0e47c372] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561a0e47fb23] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 4: (MDLog::_replay_thread()+0x617) [0x561a0e4da647] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561a0e4e3d31] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff4a932e1ca] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 7: clone() 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 0> 2024-04-23T18:18:06.475+0000 7ff497f77700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: in thread 7ff497f77700 thread_name:md_log_replay 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4a9338d20] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2: /usr/bin/ceph-mds(+0x549372) [0x561a0e47c372] 2024-04-23T18:18:06.744 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561a0e47fb23] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 4: (MDLog::_replay_thread()+0x617) [0x561a0e4da647] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561a0e4e3d31] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff4a932e1ca] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 7: clone() 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 0> 2024-04-23T18:18:06.475+0000 7ff497f77700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: in thread 7ff497f77700 thread_name:md_log_replay 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4a9338d20] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2: /usr/bin/ceph-mds(+0x549372) [0x561a0e47c372] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561a0e47fb23] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 4: (MDLog::_replay_thread()+0x617) [0x561a0e4da647] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561a0e4e3d31] 2024-04-23T18:18:06.745 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff4a932e1ca] 2024-04-23T18:18:06.746 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 7: clone() 2024-04-23T18:18:06.746 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:18:06.746 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[63439]: 2024-04-23T18:18:06.746 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:06 smithi131 podman[66644]: 2024-04-23 18:18:06.532906005 +0000 UTC m=+0.022080618 container died cf63d63b7abede0cffb50d3a4bb2b2c5f3600ba0f84e79443d444c6470af040d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:18:06.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:06 smithi110 ceph-mon[32585]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-23T18:18:06.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:06 smithi026 ceph-mon[29364]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-23T18:18:07.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 6026 23.87 MB/sec warmup 2 sec latency 11.909 ms 2024-04-23T18:18:07.741 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:07 smithi131 podman[66644]: 2024-04-23 18:18:07.267636224 +0000 UTC m=+0.756810820 container remove cf63d63b7abede0cffb50d3a4bb2b2c5f3600ba0f84e79443d444c6470af040d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-23T18:18:07.742 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:07 smithi131 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-23T18:18:07.742 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:07 smithi131 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-23T18:18:08.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 8512 21.42 MB/sec warmup 3 sec latency 11.619 ms 2024-04-23T18:18:08.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:08 smithi110 ceph-mon[32585]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-23T18:18:08.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:08 smithi026 ceph-mon[29364]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-23T18:18:08.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:08 smithi131 ceph-mon[33041]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-23T18:18:09.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 11434 19.68 MB/sec warmup 4 sec latency 15.657 ms 2024-04-23T18:18:10.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 14641 19.49 MB/sec warmup 5 sec latency 7.099 ms 2024-04-23T18:18:10.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:10 smithi110 ceph-mon[32585]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-23T18:18:10.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:10 smithi026 ceph-mon[29364]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-23T18:18:10.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:10 smithi131 ceph-mon[33041]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-23T18:18:11.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 17872 19.20 MB/sec warmup 6 sec latency 25.744 ms 2024-04-23T18:18:12.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 20697 18.45 MB/sec warmup 7 sec latency 15.438 ms 2024-04-23T18:18:12.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:12 smithi110 ceph-mon[32585]: pgmap v34: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 717 KiB/s wr, 103 op/s 2024-04-23T18:18:12.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:12 smithi026 ceph-mon[29364]: pgmap v34: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 717 KiB/s wr, 103 op/s 2024-04-23T18:18:12.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:12 smithi131 ceph-mon[33041]: pgmap v34: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 717 KiB/s wr, 103 op/s 2024-04-23T18:18:13.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 23019 17.62 MB/sec warmup 8 sec latency 48.629 ms 2024-04-23T18:18:14.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 25542 17.22 MB/sec warmup 9 sec latency 27.904 ms 2024-04-23T18:18:14.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:14 smithi110 ceph-mon[32585]: pgmap v35: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 717 KiB/s wr, 100 op/s 2024-04-23T18:18:14.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:14 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:14.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:14 smithi026 ceph-mon[29364]: pgmap v35: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 717 KiB/s wr, 100 op/s 2024-04-23T18:18:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:14 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:14.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:14 smithi131 ceph-mon[33041]: pgmap v35: 97 pgs: 97 active+clean; 8.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 717 KiB/s wr, 100 op/s 2024-04-23T18:18:14.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:14 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:15.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 28737 17.37 MB/sec warmup 10 sec latency 11.218 ms 2024-04-23T18:18:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:15 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:15 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:15 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:15 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:16.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:15 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:15 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:15 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:15 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:15 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:16.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:15 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:15 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:15 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:15 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:16.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:15 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:16.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:15 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:18:16.404 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-23T18:18:16.404 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:16.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.517+0000 7f639ca8a640 1 -- 172.21.15.26:0/331978234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63981031e0 msgr2=0x7f63981035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.517+0000 7f639ca8a640 1 --2- 172.21.15.26:0/331978234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63981031e0 0x7f63981035c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6388009900 tx=0x7f638802a510 comp rx=0 tx=0).stop 2024-04-23T18:18:16.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 -- 172.21.15.26:0/331978234 shutdown_connections 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 --2- 172.21.15.26:0/331978234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639810c520 0x7f639810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 --2- 172.21.15.26:0/331978234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f639810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 --2- 172.21.15.26:0/331978234 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f63981031e0 0x7f63981035c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 -- 172.21.15.26:0/331978234 >> 172.21.15.26:0/331978234 conn(0x7f6398077f50 msgr2=0x7f6398078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 -- 172.21.15.26:0/331978234 shutdown_connections 2024-04-23T18:18:16.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 -- 172.21.15.26:0/331978234 wait complete. 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 Processor -- start 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.518+0000 7f639ca8a640 1 -- start start 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f6398101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6398112200 con 0x7f639810c520 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6398112080 con 0x7f63981031e0 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6398112380 con 0x7f6398103b00 2024-04-23T18:18:16.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f638ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f6398101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:33096/0 (socket says 172.21.15.26:33096) 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33686/0 (socket says 172.21.15.26:33686) 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f638ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f6398101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:47776/0 (socket says 172.21.15.26:47776) 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639659c640 1 -- 172.21.15.26:0/164032174 learned_addr learned my addr 172.21.15.26:0/164032174 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 -- 172.21.15.26:0/164032174 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 msgr2=0x7f6398101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f6398101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 -- 172.21.15.26:0/164032174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 msgr2=0x7f6398100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 -- 172.21.15.26:0/164032174 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63981a6ce0 con 0x7f639810c520 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639659c640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f6396d9d640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6390009fd0 tx=0x7f639000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000ce20 con 0x7f639810c520 2024-04-23T18:18:16.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6390005020 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63981a6fd0 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.519+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390010690 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.520+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63981a74a0 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.521+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6398068d70 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.522+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f63900054d0 con 0x7f639810c520 2024-04-23T18:18:16.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.522+0000 7f638f7fe640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 0x7f63840a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.522+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f63900d10e0 con 0x7f639810c520 2024-04-23T18:18:16.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.522+0000 7f639659c640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 0x7f63840a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.523+0000 7f639659c640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 0x7f63840a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63880096c0 tx=0x7f6388005a20 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.527+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f639009d560 con 0x7f639810c520 2024-04-23T18:18:16.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 31787 17.13 MB/sec warmup 11 sec latency 13.736 ms 2024-04-23T18:18:16.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.765+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f639802d5c0 con 0x7f639810c520 2024-04-23T18:18:16.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.766+0000 7f638f7fe640 1 -- 172.21.15.26:0/164032174 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7f639009ce60 con 0x7f639810c520 2024-04-23T18:18:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:18:16.760 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:17:35.666815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2204987461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2204987461},{"type":"v1","addr":"172.21.15.131:6837","nonce":2204987461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:18:16.760 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 24 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 msgr2=0x7f63840a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 0x7f63840a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63880096c0 tx=0x7f6388005a20 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 msgr2=0x7f63981018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6390009fd0 tx=0x7f639000ef10 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 shutdown_connections 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f63840a3e40 0x7f63840a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f639810c520 0x7f63981018b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6398103b00 0x7f6398101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 --2- 172.21.15.26:0/164032174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63981031e0 0x7f6398100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 >> 172.21.15.26:0/164032174 conn(0x7f6398077f50 msgr2=0x7f63980ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 shutdown_connections 2024-04-23T18:18:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.768+0000 7f639ca8a640 1 -- 172.21.15.26:0/164032174 wait complete. 2024-04-23T18:18:16.773 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-23T18:18:16.773 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- 172.21.15.26:0/412262538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd4101a50 msgr2=0x7fbfd4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/412262538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd4101a50 0x7fbfd4105a90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009900 tx=0x7fbfc802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- 172.21.15.26:0/412262538 shutdown_connections 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/412262538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 0x7fbfd410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/412262538 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd4101a50 0x7fbfd4105a90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/412262538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd41010a0 0x7fbfd4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- 172.21.15.26:0/412262538 >> 172.21.15.26:0/412262538 conn(0x7fbfd4077f50 msgr2=0x7fbfd4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- 172.21.15.26:0/412262538 shutdown_connections 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- 172.21.15.26:0/412262538 wait complete. 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 Processor -- start 2024-04-23T18:18:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.913+0000 7fbfda0e1640 1 -- start start 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 0x7fbfd4198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd419d010 0x7fbfd419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41121f0 con 0x7fbfd41010a0 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd4112070 con 0x7fbfd419d010 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd4112370 con 0x7fbfd4106170 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33696/0 (socket says 172.21.15.26:33696) 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 -- 172.21.15.26:0/844508601 learned_addr learned my addr 172.21.15.26:0/844508601 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 0x7fbfd4198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd419d010 0x7fbfd419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 -- 172.21.15.26:0/844508601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 msgr2=0x7fbfd4198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 0x7fbfd4198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 -- 172.21.15.26:0/844508601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd419d010 msgr2=0x7fbfd419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd419d010 0x7fbfd419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 -- 172.21.15.26:0/844508601 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd419db00 con 0x7fbfd41010a0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400e9f0 tx=0x7fbfc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd0ff9640 1 -- 172.21.15.26:0/844508601 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc400cd80 con 0x7fbfd41010a0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd0ff9640 1 -- 172.21.15.26:0/844508601 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc40108e0 con 0x7fbfd41010a0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd419ddf0 con 0x7fbfd41010a0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.914+0000 7fbfd0ff9640 1 -- 172.21.15.26:0/844508601 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4018980 con 0x7fbfd41010a0 2024-04-23T18:18:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.915+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd41a5600 con 0x7fbfd41010a0 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.917+0000 7fbfd0ff9640 1 -- 172.21.15.26:0/844508601 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fbfc4010430 con 0x7fbfd41010a0 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.917+0000 7fbfdcdf9740 1 Processor -- start 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.917+0000 7fbfdcdf9740 1 -- start start 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a668a0 0x5640d3a74f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a77890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5640d39cb370 con 0x5640d3a65f80 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5640d39682e0 con 0x5640d3a668a0 2024-04-23T18:18:16.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5640d39c7330 con 0x5640d3a754a0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd0ff9640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 0x7fbfbc0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a668a0 0x5640d3a74f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33706/0 (socket says 172.21.15.26:33706) 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3396973870 learned_addr learned my addr 172.21.15.26:0/3396973870 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a77890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 0x7fbfbc0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd0ff9640 1 -- 172.21.15.26:0/844508601 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fbfc40d0db0 con 0x7fbfd41010a0 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3396973870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 msgr2=0x5640d3a77890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a77890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3396973870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a668a0 msgr2=0x5640d3a74f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a668a0 0x5640d3a74f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3396973870 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d37a9d80 con 0x5640d3a65f80 2024-04-23T18:18:16.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc011da0 tx=0x7fbfcc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bb7edaa61eddbd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 0x7fbfbc0a6380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00fc90 tx=0x7fbfcc01e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.918+0000 7fbfb9ffb640 1 -- 172.21.15.26:0/3396973870 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc022070 con 0x5640d3a65f80 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfb9ffb640 1 -- 172.21.15.26:0/3396973870 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfcc009dc0 con 0x5640d3a65f80 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfb9ffb640 1 -- 172.21.15.26:0/3396973870 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc004050 con 0x5640d3a65f80 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3396973870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 msgr2=0x5640d3a66360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc011da0 tx=0x7fbfcc00f790 comp rx=0 tx=0).stop 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3396973870 shutdown_connections 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a668a0 0x5640d3a74f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a77890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3396973870 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a66360 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3396973870 >> 172.21.15.26:0/3396973870 conn(0x5640d3a623c0 msgr2=0x5640d3a627a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3396973870 shutdown_connections 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.919+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3396973870 wait complete. 2024-04-23T18:18:16.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 -- start start 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 Processor -- start 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a668a0 0x5640d3a729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a72f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5640d3a78280 con 0x5640d3a65f80 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5640d3965fb0 con 0x5640d3a754a0 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5640d3a78400 con 0x5640d3a668a0 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfdcdf9740 10 client.0 _ll_register_callbacks cb 0x5640d39659a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a72f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33716/0 (socket says 172.21.15.26:33716) 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfd37fe640 1 -- 172.21.15.26:0/3091890611 learned_addr learned my addr 172.21.15.26:0/3091890611 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.920+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a668a0 0x5640d3a729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3091890611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a668a0 msgr2=0x5640d3a729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a668a0 0x5640d3a729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3091890611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 msgr2=0x5640d3a72490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 -- 172.21.15.26:0/3091890611 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d380a510 con 0x5640d3a754a0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a72f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0099b0 tx=0x7fbfcc011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61116775f726cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc022030 con 0x5640d3a754a0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfcc009dc0 con 0x5640d3a754a0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fbfcc005000 con 0x5640d3a754a0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfcc03c050 con 0x5640d3a754a0 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfdcdf9740 10 client.25235 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:16.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.921+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640d369d670 con 0x5640d3a754a0 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfcc009dc0 con 0x5640d3a754a0 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfcc02e540 con 0x5640d3a754a0 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 10 client.25235 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 10 client.25235 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 20 client.25235 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3a858a0 0x5640d3a87c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 4 client.25235 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d39d8d90 con 0x5640d3a858a0 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3a858a0 0x5640d3a87c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfb2ffd640 10 client.25235 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:16.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.922+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3a858a0 0x5640d3a87c90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed63f958c7a91d5 server_cookie=1058ca05290b46dd in_seq=0 out_seq=0 2024-04-23T18:18:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/3091890611 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640d39d8d90 con 0x5640d3a858a0 2024-04-23T18:18:16.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfb2ffd640 10 client.25235 handle_command_reply: tid=0 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 client.25235 shutdown 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 msgr2=0x5640d3a72f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a72f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0099b0 tx=0x7fbfcc011aa0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 shutdown_connections 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3a858a0 0x5640d3a87c90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a72f10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a668a0 0x5640d3a729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfb2ffd640 0 client.25235 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/3091890611 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a65f80 0x5640d3a72490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.923+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 >> 172.21.15.26:0/3091890611 conn(0x5640d3a623c0 msgr2=0x5640d3a68150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.924+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 shutdown_connections 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.924+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/3091890611 wait complete. 2024-04-23T18:18:16.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.924+0000 7fbfdcdf9740 20 client.25235 trim_cache size 0 max 16384 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 Processor -- start 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 -- start start 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a65f80 0x5640d3a81180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a8d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41ae1c0 con 0x5640d3a668a0 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41aba90 con 0x5640d3a65f80 2024-04-23T18:18:16.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41a9900 con 0x5640d3a754a0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33732/0 (socket says 172.21.15.26:33732) 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/106595041 learned_addr learned my addr 172.21.15.26:0/106595041 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a65f80 0x5640d3a81180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.941+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a8d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/106595041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 msgr2=0x5640d3a8d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a8d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/106595041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a65f80 msgr2=0x5640d3a81180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a65f80 0x5640d3a81180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/106595041 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d3a88000 con 0x5640d3a668a0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8033000 tx=0x7fbfc8032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efddc3c3d15b4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfb97fa640 1 -- 172.21.15.26:0/106595041 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035040 con 0x5640d3a668a0 2024-04-23T18:18:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfb97fa640 1 -- 172.21.15.26:0/106595041 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc80316a0 con 0x5640d3a668a0 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.942+0000 7fbfb97fa640 1 -- 172.21.15.26:0/106595041 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8004750 con 0x5640d3a668a0 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/106595041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 msgr2=0x5640d3a743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8033000 tx=0x7fbfc8032df0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/106595041 shutdown_connections 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a754a0 0x5640d3a8d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a743d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/106595041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a65f80 0x5640d3a81180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/106595041 >> 172.21.15.26:0/106595041 conn(0x5640d3a623c0 msgr2=0x5640d3a73cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/106595041 shutdown_connections 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/106595041 wait complete. 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 -- start start 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.943+0000 7fbfdcdf9740 1 Processor -- start 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a65f80 0x5640d3a7d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a79170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41a9f50 con 0x5640d3a668a0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41ab3c0 con 0x5640d3a754a0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41aa620 con 0x5640d3a65f80 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 10 client.0 _ll_register_callbacks cb 0x7fbfcc011ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33742/0 (socket says 172.21.15.26:33742) 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/1595890792 learned_addr learned my addr 172.21.15.26:0/1595890792 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd3fff640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a79170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a65f80 0x5640d3a7d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/1595890792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a65f80 msgr2=0x5640d3a7d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a65f80 0x5640d3a7d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 msgr2=0x5640d3a79170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a79170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 -- 172.21.15.26:0/1595890792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d3a797b0 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfd2ffd640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfc802a450 tx=0x7fbfc8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a09f82dc109b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035030 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fbfc8005520 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfc80316a0 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfc8057050 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.944+0000 7fbfdcdf9740 10 client.15708 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640d3aebbd0 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc80316a0 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfc8004960 con 0x5640d3a668a0 2024-04-23T18:18:16.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 10 client.15708 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 10 client.15708 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 20 client.15708 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3aec3a0 0x5640d3aee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 4 client.15708 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x5640d39d2c00 con 0x5640d3aec3a0 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.945+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3aec3a0 0x5640d3aee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.946+0000 7fbfd37fe640 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3aec3a0 0x5640d3aee7c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a0bca8c104d20 server_cookie=fba7183d3fb84e8c in_seq=0 out_seq=0 2024-04-23T18:18:16.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:16.946+0000 7fbfb2ffd640 10 client.15708 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfb2ffd640 1 -- 172.21.15.26:0/1595890792 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5640d39d2c00 con 0x5640d3aec3a0 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfb2ffd640 10 client.15708 handle_command_reply: tid=0 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 client.15708 shutdown 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 msgr2=0x5640d3a78c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfc802a450 tx=0x7fbfc8005850 comp rx=0 tx=0).stop 2024-04-23T18:18:17.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 shutdown_connections 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5640d3aec3a0 0x5640d3aee7c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640d3a754a0 0x5640d3a79170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5640d3a668a0 0x5640d3a78c30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfb2ffd640 0 client.15708 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 --2- 172.21.15.26:0/1595890792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5640d3a65f80 0x5640d3a7d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.089+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 >> 172.21.15.26:0/1595890792 conn(0x5640d3a623c0 msgr2=0x5640d3a74550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 shutdown_connections 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfdcdf9740 1 -- 172.21.15.26:0/1595890792 wait complete. 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfdcdf9740 20 client.15708 trim_cache size 0 max 16384 2024-04-23T18:18:17.084 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 24, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 6, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "1ea84704": { 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 6, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:18:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:17.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 msgr2=0x7fbfbc0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 0x7fbfbc0a6380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00fc90 tx=0x7fbfcc01e000 comp rx=0 tx=0).stop 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 msgr2=0x7fbfd4198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400e9f0 tx=0x7fbfc400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 shutdown_connections 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbfbc0a3ec0 0x7fbfbc0a6380 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd419d010 0x7fbfd419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfd4106170 0x7fbfd4198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 --2- 172.21.15.26:0/844508601 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbfd41010a0 0x7fbfd4198750 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 >> 172.21.15.26:0/844508601 conn(0x7fbfd4077f50 msgr2=0x7fbfd40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:17.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 shutdown_connections 2024-04-23T18:18:17.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:17.090+0000 7fbfda0e1640 1 -- 172.21.15.26:0/844508601 wait complete. 2024-04-23T18:18:17.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:16 smithi026 ceph-mon[29364]: pgmap v36: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-04-23T18:18:17.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/164032174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:17.094 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1ea84704', quiesce took 0.7 seconds. Will release after: 10 2024-04-23T18:18:17.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:16 smithi110 ceph-mon[32585]: pgmap v36: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-04-23T18:18:17.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/164032174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:17.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:16 smithi131 ceph-mon[33041]: pgmap v36: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 183 op/s 2024-04-23T18:18:17.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/164032174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:17.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 16.09 MB/sec warmup 12 sec latency 573.755 ms 2024-04-23T18:18:18.092 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:17 smithi131 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-23T18:18:18.092 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:17 smithi131 systemd[1]: Stopped Ceph mds.c for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:18:18.092 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:17 smithi131 systemd[1]: Starting Ceph mds.c for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:18:18.092 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:17 smithi131 podman[66868]: 2024-04-23 18:18:17.88355077 +0000 UTC m=+0.017942933 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:18:18.092 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:17 smithi131 podman[66868]: 2024-04-23 18:18:17.985616187 +0000 UTC m=+0.120008371 container create 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-23T18:18:18.349 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 podman[66868]: 2024-04-23 18:18:18.221616721 +0000 UTC m=+0.356008860 container init 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T18:18:18.349 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 podman[66868]: 2024-04-23 18:18:18.225011608 +0000 UTC m=+0.359403748 container start 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T18:18:18.349 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 ceph-mds[66888]: -- 172.21.15.131:0/3108198777 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5646eec3fa40 con 0x5646eec12c00 2024-04-23T18:18:18.349 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: starting mds.c at 2024-04-23T18:18:18.350 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 bash[66868]: 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b 2024-04-23T18:18:18.350 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:18:18 smithi131 systemd[1]: Started Ceph mds.c for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:18:18.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 14.85 MB/sec warmup 13 sec latency 1573.851 ms 2024-04-23T18:18:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 -- 172.21.15.26:0/429807411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0104d50 msgr2=0x7f7da0108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 --2- 172.21.15.26:0/429807411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0104d50 0x7f7da0108d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009900 tx=0x7f7d8802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 -- 172.21.15.26:0/429807411 shutdown_connections 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 --2- 172.21.15.26:0/429807411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0104d50 0x7f7da0108d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 --2- 172.21.15.26:0/429807411 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da0109470 0x7f7da0111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 --2- 172.21.15.26:0/429807411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da01043a0 0x7f7da0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.770+0000 7f7da50ef640 1 -- 172.21.15.26:0/429807411 >> 172.21.15.26:0/429807411 conn(0x7f7da00ffc40 msgr2=0x7f7da0102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.771+0000 7f7da50ef640 1 -- 172.21.15.26:0/429807411 shutdown_connections 2024-04-23T18:18:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.771+0000 7f7da50ef640 1 -- 172.21.15.26:0/429807411 wait complete. 2024-04-23T18:18:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.771+0000 7f7da50ef640 1 Processor -- start 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 -- start start 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0109470 0x7f7da01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 0x7f7da019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0114390 con 0x7f7da01043a0 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0114210 con 0x7f7da019a730 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7da50ef640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0114510 con 0x7f7da0109470 2024-04-23T18:18:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43310/0 (socket says 172.21.15.26:43310) 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/1630492916 learned_addr learned my addr 172.21.15.26:0/1630492916 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0109470 0x7f7da01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 0x7f7da019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/1630492916 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0109470 msgr2=0x7f7da01a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0109470 0x7f7da01a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/1630492916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 msgr2=0x7f7da019abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 0x7f7da019abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/1630492916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da019b330 con 0x7f7da01043a0 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 0x7f7da019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:18.769 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.772+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.773+0000 7f7d97fff640 1 -- 172.21.15.26:0/1630492916 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000cd80 con 0x7f7da01043a0 2024-04-23T18:18:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.773+0000 7f7d97fff640 1 -- 172.21.15.26:0/1630492916 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d900108e0 con 0x7f7da01043a0 2024-04-23T18:18:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.773+0000 7f7d97fff640 1 -- 172.21.15.26:0/1630492916 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90018980 con 0x7f7da01043a0 2024-04-23T18:18:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.773+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da01a73c0 con 0x7f7da01043a0 2024-04-23T18:18:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.773+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da01a7770 con 0x7f7da01043a0 2024-04-23T18:18:18.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.775+0000 7f7d97fff640 1 -- 172.21.15.26:0/1630492916 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f7d90010430 con 0x7f7da01043a0 2024-04-23T18:18:18.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.775+0000 7f7da7e06740 1 Processor -- start 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.775+0000 7f7da7e06740 1 -- start start 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809ce620 0x55ce809cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809d92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 0x55ce809dbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ce80933970 con 0x55ce809cef40 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce808d04a0 con 0x55ce809ce620 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7da7e06740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ce8092f930 con 0x55ce809d9800 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d97fff640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 0x7f7d8c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809d92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 0x55ce809dbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809d92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43324/0 (socket says 172.21.15.26:43324) 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/2309124961 learned_addr learned my addr 172.21.15.26:0/2309124961 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 0x7f7d8c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809ce620 0x55ce809cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d97fff640 1 -- 172.21.15.26:0/1630492916 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7d900d0e50 con 0x7f7da01043a0 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2309124961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809ce620 msgr2=0x55ce809cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809ce620 0x55ce809cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.776+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2309124961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 msgr2=0x55ce809d92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809d92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2309124961 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce80646210 con 0x55ce809d9800 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 0x55ce809dbbf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006010 tx=0x7f7d88005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dc060a40b128a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 0x7f7d8c0a63a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009fd0 tx=0x7f7d98009b30 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d957fa640 1 -- 172.21.15.26:0/2309124961 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88040070 con 0x55ce809d9800 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d957fa640 1 -- 172.21.15.26:0/2309124961 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d88004030 con 0x55ce809d9800 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.777+0000 7f7d957fa640 1 -- 172.21.15.26:0/2309124961 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88035040 con 0x55ce809d9800 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- 172.21.15.26:0/2309124961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 msgr2=0x55ce809dbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 0x55ce809dbbf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d88006010 tx=0x7f7d88005e40 comp rx=0 tx=0).stop 2024-04-23T18:18:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- 172.21.15.26:0/2309124961 shutdown_connections 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809d9800 0x55ce809dbbf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809d92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2309124961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809ce620 0x55ce809cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- 172.21.15.26:0/2309124961 >> 172.21.15.26:0/2309124961 conn(0x55ce809cab30 msgr2=0x55ce809caf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- 172.21.15.26:0/2309124961 shutdown_connections 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- 172.21.15.26:0/2309124961 wait complete. 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 -- start start 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.778+0000 7f7da7e06740 1 Processor -- start 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809d91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809cef40 0x55ce809d64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ce809dc3d0 con 0x55ce809d9800 2024-04-23T18:18:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce808ce0b0 con 0x55ce809cef40 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ce809dc550 con 0x55ce809ce620 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7da7e06740 10 client.0 _ll_register_callbacks cb 0x55ce808bbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809d91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43336/0 (socket says 172.21.15.26:43336) 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 -- 172.21.15.26:0/1789584005 learned_addr learned my addr 172.21.15.26:0/1789584005 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809cef40 0x55ce809d64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 -- 172.21.15.26:0/1789584005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 msgr2=0x55ce809d91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809d91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 -- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809cef40 msgr2=0x55ce809d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809cef40 0x55ce809d64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 -- 172.21.15.26:0/1789584005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce80712620 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809d91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7d98010b40 tx=0x7f7d98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5cbbbc9ce1c26b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d867fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.779+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98012e60 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7d98011bd0 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d88004030 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7d98011dd0 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 10 client.15723 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce809e83c0 con 0x55ce809d9800 2024-04-23T18:18:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d88004030 con 0x55ce809d9800 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7d98026070 con 0x55ce809d9800 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 10 client.15723 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 20 client.15723 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce809e9940 0x55ce809ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 4 client.15723 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce80941390 con 0x55ce809e9940 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.780+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce809e9940 0x55ce809ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.781+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce809e9940 0x55ce809ebd50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9824c9e8ce2b0ea server_cookie=8afc03b222bf818f in_seq=0 out_seq=0 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.781+0000 7f7d867fc640 10 client.15723 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7d867fc640 1 -- 172.21.15.26:0/1789584005 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce80941390 con 0x55ce809e9940 2024-04-23T18:18:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7d867fc640 10 client.15723 handle_command_reply: tid=0 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 client.15723 shutdown 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 msgr2=0x55ce809d6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7d98010b40 tx=0x7f7d98004ab0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 shutdown_connections 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce809e9940 0x55ce809ebd50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809cef40 0x55ce809d64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809d9800 0x55ce809d6ac0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 --2- 172.21.15.26:0/1789584005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809d91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7d867fc640 0 client.15723 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 >> 172.21.15.26:0/1789584005 conn(0x55ce809cab30 msgr2=0x55ce809d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 shutdown_connections 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 1 -- 172.21.15.26:0/1789584005 wait complete. 2024-04-23T18:18:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.782+0000 7f7da7e06740 20 client.15723 trim_cache size 0 max 16384 2024-04-23T18:18:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 Processor -- start 2024-04-23T18:18:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 -- start start 2024-04-23T18:18:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 0x55ce809d5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce809c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01b24e0 con 0x55ce809ce620 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01afdb0 con 0x55ce809d9800 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7da7e06740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01adc20 con 0x55ce809cef40 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43342/0 (socket says 172.21.15.26:43342) 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/2538094422 learned_addr learned my addr 172.21.15.26:0/2538094422 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.797+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 0x55ce809d5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce809c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/2538094422 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 msgr2=0x55ce809d5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 0x55ce809d5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/2538094422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 msgr2=0x55ce809c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce809c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 -- 172.21.15.26:0/2538094422 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce809ca8a0 con 0x55ce809ce620 2024-04-23T18:18:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 0x55ce809d5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7d900994f0 tx=0x7f7d9001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8084415d53afcb40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d94ff9640 1 -- 172.21.15.26:0/2538094422 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90026040 con 0x55ce809ce620 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d94ff9640 1 -- 172.21.15.26:0/2538094422 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d900a0f20 con 0x55ce809ce620 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.798+0000 7f7d94ff9640 1 -- 172.21.15.26:0/2538094422 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9009c220 con 0x55ce809ce620 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- 172.21.15.26:0/2538094422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 msgr2=0x55ce809e5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7d900994f0 tx=0x7f7d9001a040 comp rx=0 tx=0).stop 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- 172.21.15.26:0/2538094422 shutdown_connections 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce809c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809cef40 0x55ce809d5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2538094422 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809ce620 0x55ce809e5290 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- 172.21.15.26:0/2538094422 >> 172.21.15.26:0/2538094422 conn(0x55ce809cab30 msgr2=0x55ce809c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- 172.21.15.26:0/2538094422 shutdown_connections 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- 172.21.15.26:0/2538094422 wait complete. 2024-04-23T18:18:18.794 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- start start 2024-04-23T18:18:18.795 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 Processor -- start 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809dea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce80a47f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ae270 con 0x55ce809cef40 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01af6e0 con 0x55ce809d9800 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ae940 con 0x55ce809ce620 2024-04-23T18:18:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.799+0000 7f7da7e06740 10 client.0 _ll_register_callbacks cb 0x55ce808bbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43350/0 (socket says 172.21.15.26:43350) 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2496764987 learned_addr learned my addr 172.21.15.26:0/2496764987 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809dea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce80a47f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2496764987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 msgr2=0x55ce809dea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809dea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 msgr2=0x55ce80a47f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce80a47f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 -- 172.21.15.26:0/2496764987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce80a484d0 con 0x55ce809cef40 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9f59e640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce80a47f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d9e59c640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009900 tx=0x7f7d88032e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168ca1e07b7d052d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d867fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88040040 con 0x55ce809cef40 2024-04-23T18:18:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7d88004030 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d900a0f20 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7d88055050 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7da7e06740 10 client.15729 fetch_fsmap learned FSMap version 24 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.800+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ce80a4f8c0 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d900a0f20 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7d880046c0 con 0x55ce809cef40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 10 client.15729 fetch_fsmap finished waiting for FSMap version 24 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 20 client.15729 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce80a4fd40 0x55ce80a52130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 4 client.15729 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce8093b200 con 0x55ce80a4fd40 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.801+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce80a4fd40 0x55ce80a52130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.802+0000 7f7d867fc640 10 client.15729 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.802+0000 7f7d9ed9d640 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce80a4fd40 0x55ce80a52130 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71694ccdaae6131 server_cookie=b069fe59bbce1484 in_seq=0 out_seq=0 2024-04-23T18:18:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7d867fc640 1 -- 172.21.15.26:0/2496764987 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce8093b200 con 0x55ce80a4fd40 2024-04-23T18:18:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7d867fc640 10 client.15729 handle_command_reply: tid=0 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 client.15729 shutdown 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 msgr2=0x55ce809defa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009900 tx=0x7f7d88032e50 comp rx=0 tx=0).stop 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 shutdown_connections 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ce80a4fd40 0x55ce80a52130 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce809d9800 0x55ce80a47f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ce809cef40 0x55ce809defa0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7d867fc640 0 client.15729 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 --2- 172.21.15.26:0/2496764987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce809ce620 0x55ce809dea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.808+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 >> 172.21.15.26:0/2496764987 conn(0x55ce809cab30 msgr2=0x55ce809d54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 shutdown_connections 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da7e06740 1 -- 172.21.15.26:0/2496764987 wait complete. 2024-04-23T18:18:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da7e06740 20 client.15729 trim_cache size 0 max 16384 2024-04-23T18:18:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:18:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:18:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:18:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:18:18.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 msgr2=0x7f7d8c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 0x7f7d8c0a63a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009fd0 tx=0x7f7d98009b30 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 msgr2=0x7f7da01a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 shutdown_connections 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7d8c0a3ee0 0x7f7d8c0a63a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da019a730 0x7f7da019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7da0109470 0x7f7da01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 --2- 172.21.15.26:0/1630492916 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7da01043a0 0x7f7da01a0620 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 >> 172.21.15.26:0/1630492916 conn(0x7f7da00ffc40 msgr2=0x7f7da0100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 shutdown_connections 2024-04-23T18:18:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:18.809+0000 7f7da50ef640 1 -- 172.21.15.26:0/1630492916 wait complete. 2024-04-23T18:18:19.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:18 smithi110 ceph-mon[32585]: pgmap v37: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-23T18:18:19.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:18 smithi110 ceph-mon[32585]: daemon mds.c restarted 2024-04-23T18:18:19.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:18 smithi026 ceph-mon[29364]: pgmap v37: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-23T18:18:19.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:18 smithi026 ceph-mon[29364]: daemon mds.c restarted 2024-04-23T18:18:19.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:18 smithi131 ceph-mon[33041]: pgmap v37: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-23T18:18:19.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:18 smithi131 ceph-mon[33041]: daemon mds.c restarted 2024-04-23T18:18:19.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 13.79 MB/sec warmup 14 sec latency 2573.963 ms 2024-04-23T18:18:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:19 smithi110 ceph-mon[32585]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:19 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6836/2307422363,v1:172.21.15.131:6837/2307422363] up:boot 2024-04-23T18:18:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:19 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:18:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:19 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:19 smithi026 ceph-mon[29364]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:19 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6836/2307422363,v1:172.21.15.131:6837/2307422363] up:boot 2024-04-23T18:18:20.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:19 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:18:20.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:20.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:19 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:19 smithi131 ceph-mon[33041]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:19 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6836/2307422363,v1:172.21.15.131:6837/2307422363] up:boot 2024-04-23T18:18:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:19 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:18:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:20.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:19 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:20.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 12.87 MB/sec warmup 15 sec latency 3574.001 ms 2024-04-23T18:18:21.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:20 smithi110 ceph-mon[32585]: pgmap v39: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-23T18:18:21.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:20 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:20 smithi026 ceph-mon[29364]: pgmap v39: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-23T18:18:21.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:20 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:20 smithi131 ceph-mon[33041]: pgmap v39: 97 pgs: 97 active+clean; 15 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-23T18:18:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:20 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:21.656 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 12.07 MB/sec warmup 16 sec latency 4574.039 ms 2024-04-23T18:18:22.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 11.36 MB/sec warmup 17 sec latency 5574.122 ms 2024-04-23T18:18:23.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:22 smithi110 ceph-mon[32585]: pgmap v40: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:23.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:22 smithi026 ceph-mon[29364]: pgmap v40: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:22 smithi131 ceph-mon[33041]: pgmap v40: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:23.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 10.73 MB/sec warmup 18 sec latency 6574.227 ms 2024-04-23T18:18:24.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 10.16 MB/sec warmup 19 sec latency 7574.321 ms 2024-04-23T18:18:25.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:24 smithi110 ceph-mon[32585]: pgmap v41: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:25.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:24 smithi026 ceph-mon[29364]: pgmap v41: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:25.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:24 smithi131 ceph-mon[33041]: pgmap v41: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-23T18:18:25.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 9.65 MB/sec warmup 20 sec latency 8574.403 ms 2024-04-23T18:18:26.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32341 9.19 MB/sec warmup 21 sec latency 9574.480 ms 2024-04-23T18:18:27.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:27.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:26 smithi110 ceph-mon[32585]: pgmap v42: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:27.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:26 smithi026 ceph-mon[29364]: pgmap v42: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 -- 172.21.15.26:0/2748030471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c0107280 msgr2=0x7fe4c010ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/2748030471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c0107280 0x7fe4c010ee50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009ca0 tx=0x7fe4b802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 -- 172.21.15.26:0/2748030471 shutdown_connections 2024-04-23T18:18:27.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/2748030471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102bf0 0x7fe4c0106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/2748030471 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c0107280 0x7fe4c010ee50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/2748030471 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0102240 0x7fe4c0102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 -- 172.21.15.26:0/2748030471 >> 172.21.15.26:0/2748030471 conn(0x7fe4c00790e0 msgr2=0x7fe4c00794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 -- 172.21.15.26:0/2748030471 shutdown_connections 2024-04-23T18:18:27.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 -- 172.21.15.26:0/2748030471 wait complete. 2024-04-23T18:18:27.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.217+0000 7fe4c86a3640 1 Processor -- start 2024-04-23T18:18:27.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 -- start start 2024-04-23T18:18:27.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 0x7fe4c0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0107280 0x7fe4c0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0112200 con 0x7fe4c019cfe0 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0112080 con 0x7fe4c0102240 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c86a3640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0112380 con 0x7fe4c0107280 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c6c40640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c6c40640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43380/0 (socket says 172.21.15.26:43380) 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c6c40640 1 -- 172.21.15.26:0/755997481 learned_addr learned my addr 172.21.15.26:0/755997481 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c643f640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 0x7fe4c0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c5c3e640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0107280 0x7fe4c0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.218+0000 7fe4c6c40640 1 -- 172.21.15.26:0/755997481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0107280 msgr2=0x7fe4c0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c6c40640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0107280 0x7fe4c0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c6c40640 1 -- 172.21.15.26:0/755997481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 msgr2=0x7fe4c0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c6c40640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 0x7fe4c0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c6c40640 1 -- 172.21.15.26:0/755997481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c019dad0 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c643f640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 0x7fe4c0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c6c40640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80098d0 tx=0x7fe4b80317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8003b70 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4b8003d10 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c019dd60 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b802fa20 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.219+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c01a55b0 con 0x7fe4c019cfe0 2024-04-23T18:18:27.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.220+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c0199a70 con 0x7fe4c019cfe0 2024-04-23T18:18:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.221+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fe4b802fc20 con 0x7fe4c019cfe0 2024-04-23T18:18:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.222+0000 7fe4b77fe640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 0x7fe4ac0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.222+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe4b80f1fc0 con 0x7fe4c019cfe0 2024-04-23T18:18:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.222+0000 7fe4c643f640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 0x7fe4ac0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.222+0000 7fe4c643f640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 0x7fe4ac0a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005fd0 tx=0x7fe4bc00b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.226+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4b80be550 con 0x7fe4c019cfe0 2024-04-23T18:18:27.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:26 smithi131 ceph-mon[33041]: pgmap v42: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:27.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.450+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4c00691a0 con 0x7fe4c019cfe0 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.450+0000 7fe4b77fe640 1 -- 172.21.15.26:0/755997481 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11888 (secure 0 0 0) 0x7fe4c00691a0 con 0x7fe4c019cfe0 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:18:27.445 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:18:19.408028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:18:27.446 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 27 2024-04-23T18:18:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.452+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 msgr2=0x7fe4ac0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.452+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 0x7fe4ac0a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005fd0 tx=0x7fe4bc00b390 comp rx=0 tx=0).stop 2024-04-23T18:18:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.452+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 msgr2=0x7fe4c019d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.452+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80098d0 tx=0x7fe4b80317b0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 shutdown_connections 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe4ac0a3ec0 0x7fe4ac0a6380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe4c019cfe0 0x7fe4c019d490 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4c0107280 0x7fe4c0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 --2- 172.21.15.26:0/755997481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c0102240 0x7fe4c0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 >> 172.21.15.26:0/755997481 conn(0x7fe4c00790e0 msgr2=0x7fe4c01002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 shutdown_connections 2024-04-23T18:18:27.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.453+0000 7fe4c86a3640 1 -- 172.21.15.26:0/755997481 wait complete. 2024-04-23T18:18:27.458 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 1ea84704 --release --await' 2024-04-23T18:18:27.458 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 1ea84704 --release --await 2024-04-23T18:18:27.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 -- 172.21.15.26:0/1849459616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc198101a50 msgr2=0x7fc198105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 --2- 172.21.15.26:0/1849459616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc198101a50 0x7fc198105a90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009900 tx=0x7fc18c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 -- 172.21.15.26:0/1849459616 shutdown_connections 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 --2- 172.21.15.26:0/1849459616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 0x7fc19810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 --2- 172.21.15.26:0/1849459616 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc198101a50 0x7fc198105a90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 --2- 172.21.15.26:0/1849459616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981010a0 0x7fc198101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 -- 172.21.15.26:0/1849459616 >> 172.21.15.26:0/1849459616 conn(0x7fc198077eb0 msgr2=0x7fc1980782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 -- 172.21.15.26:0/1849459616 shutdown_connections 2024-04-23T18:18:27.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.573+0000 7fc19cd30640 1 -- 172.21.15.26:0/1849459616 wait complete. 2024-04-23T18:18:27.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 Processor -- start 2024-04-23T18:18:27.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 -- start start 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 0x7fc198103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981035b0 0x7fc1981a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981110b0 con 0x7fc1981010a0 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc198110f30 con 0x7fc198106170 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19cd30640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc198111230 con 0x7fc1981035b0 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43404/0 (socket says 172.21.15.26:43404) 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 -- 172.21.15.26:0/3174687800 learned_addr learned my addr 172.21.15.26:0/3174687800 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc195d9b640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 0x7fc198103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981035b0 0x7fc1981a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 -- 172.21.15.26:0/3174687800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981035b0 msgr2=0x7fc1981a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981035b0 0x7fc1981a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 -- 172.21.15.26:0/3174687800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 msgr2=0x7fc198103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 0x7fc198103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 -- 172.21.15.26:0/3174687800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1981a7010 con 0x7fc1981010a0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.574+0000 7fc19659c640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc18800d770 tx=0x7fc18800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.575+0000 7fc1877fe640 1 -- 172.21.15.26:0/3174687800 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188017070 con 0x7fc1981010a0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.575+0000 7fc1877fe640 1 -- 172.21.15.26:0/3174687800 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1880054d0 con 0x7fc1981010a0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.575+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1981a7300 con 0x7fc1981010a0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.575+0000 7fc1877fe640 1 -- 172.21.15.26:0/3174687800 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188004c30 con 0x7fc1981010a0 2024-04-23T18:18:27.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.575+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1981a77d0 con 0x7fc1981010a0 2024-04-23T18:18:27.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc1877fe640 1 -- 172.21.15.26:0/3174687800 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fc188004dd0 con 0x7fc1981010a0 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 Processor -- start 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 -- start start 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846e7480 0x5584846e9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5584846416d0 con 0x5584846dc490 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5584845de1a0 con 0x5584846dcdb0 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.577+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55848463d690 con 0x5584846e7480 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43416/0 (socket says 172.21.15.26:43416) 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 -- 172.21.15.26:0/2226827302 learned_addr learned my addr 172.21.15.26:0/2226827302 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc19659c640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc1877fe640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 0x7fc1800a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc195d9b640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846e7480 0x5584846e9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 -- 172.21.15.26:0/2226827302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846e7480 msgr2=0x5584846e9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846e7480 0x5584846e9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 -- 172.21.15.26:0/2226827302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 msgr2=0x5584846e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 -- 172.21.15.26:0/2226827302 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584845e85b0 con 0x5584846dc490 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc19659c640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 0x7fc1800a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc1877fe640 1 -- 172.21.15.26:0/3174687800 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc1880d0540 con 0x7fc1981010a0 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc196d9d640 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc190010520 tx=0x7fc1900109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586a7f528df166a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc184ff9640 1 -- 172.21.15.26:0/2226827302 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190019070 con 0x5584846dc490 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc184ff9640 1 -- 172.21.15.26:0/2226827302 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc190003e10 con 0x5584846dc490 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.578+0000 7fc184ff9640 1 -- 172.21.15.26:0/2226827302 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1900116a0 con 0x5584846dc490 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 -- 172.21.15.26:0/2226827302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 msgr2=0x5584846dc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc190010520 tx=0x7fc1900109e0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 0x7fc1800a6330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc190021000 tx=0x7fc19001e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 -- 172.21.15.26:0/2226827302 shutdown_connections 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846e7480 0x5584846e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/2226827302 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846dc870 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 -- 172.21.15.26:0/2226827302 >> 172.21.15.26:0/2226827302 conn(0x5584846d87b0 msgr2=0x5584846d8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 -- 172.21.15.26:0/2226827302 shutdown_connections 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.579+0000 7fc19fa4a740 1 -- 172.21.15.26:0/2226827302 wait complete. 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 -- start start 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 Processor -- start 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 0x5584846e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5584846ea310 con 0x5584846e7480 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5584845dbb20 con 0x5584846dc490 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5584846ea490 con 0x5584846dcdb0 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19fa4a740 10 client.0 _ll_register_callbacks cb 0x5584845db530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43430/0 (socket says 172.21.15.26:43430) 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 -- 172.21.15.26:0/3578409402 learned_addr learned my addr 172.21.15.26:0/3578409402 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19659c640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 0x5584846e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc195d9b640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 -- 172.21.15.26:0/3578409402 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 msgr2=0x5584846e41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846e41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 -- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 msgr2=0x5584846e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 0x5584846e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc196d9d640 1 -- 172.21.15.26:0/3578409402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558484353920 con 0x5584846e7480 2024-04-23T18:18:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.580+0000 7fc19659c640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 0x5584846e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc190009b90 tx=0x7fc190011670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f876fb8cb56a42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190019070 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc190011c90 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc190003e10 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc190013520 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 10 client.15747 fetch_fsmap learned FSMap version 27 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55848441fd30 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc190003e10 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fc19002e070 con 0x5584846e7480 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 10 client.15747 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 10 client.15747 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 20 client.15747 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5584846f7550 0x5584846f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 4 client.15747 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.581+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584845ff500 con 0x5584846f7550 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.582+0000 7fc19659c640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5584846f7550 0x5584846f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.582+0000 7fc17dffb640 10 client.15747 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.582+0000 7fc19659c640 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5584846f7550 0x5584846f99c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db841cf00f029f08 server_cookie=eae293c45720075f in_seq=0 out_seq=0 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc17dffb640 1 -- 172.21.15.26:0/3578409402 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584845ff500 con 0x5584846f7550 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc17dffb640 10 client.15747 handle_command_reply: tid=0 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 client.15747 shutdown 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 msgr2=0x5584846e48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc190009b90 tx=0x7fc190011670 comp rx=0 tx=0).stop 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 shutdown_connections 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5584846f7550 0x5584846f99c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846e48f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846e41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3578409402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dc490 0x5584846e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc17dffb640 0 client.15747 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:27.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 >> 172.21.15.26:0/3578409402 conn(0x5584846d87b0 msgr2=0x5584846de5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 shutdown_connections 2024-04-23T18:18:27.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3578409402 wait complete. 2024-04-23T18:18:27.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.583+0000 7fc19fa4a740 20 client.15747 trim_cache size 0 max 16384 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.600+0000 7fc19fa4a740 1 Processor -- start 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.600+0000 7fc19fa4a740 1 -- start start 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.600+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846ff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846e7480 0x5584846ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981b2600 con 0x5584846dc490 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981afe40 con 0x5584846e7480 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981adcb0 con 0x5584846dcdb0 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43436/0 (socket says 172.21.15.26:43436) 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 -- 172.21.15.26:0/3982957096 learned_addr learned my addr 172.21.15.26:0/3982957096 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846e7480 0x5584846ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 -- 172.21.15.26:0/3982957096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 msgr2=0x5584846ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 -- 172.21.15.26:0/3982957096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846e7480 msgr2=0x5584846ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846e7480 0x5584846ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 -- 172.21.15.26:0/3982957096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584846dcbe0 con 0x5584846dc490 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc19659c640 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc18809c250 tx=0x7fc188009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c47e706b80324c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc17ffff640 1 -- 172.21.15.26:0/3982957096 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188020040 con 0x5584846dc490 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc17ffff640 1 -- 172.21.15.26:0/3982957096 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1880a0670 con 0x5584846dc490 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.601+0000 7fc17ffff640 1 -- 172.21.15.26:0/3982957096 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1880a4aa0 con 0x5584846dc490 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3982957096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 msgr2=0x5584846e6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc18809c250 tx=0x7fc188009f90 comp rx=0 tx=0).stop 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3982957096 shutdown_connections 2024-04-23T18:18:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846e7480 0x5584846ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dcdb0 0x5584846ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3982957096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846dc490 0x5584846e6a10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3982957096 >> 172.21.15.26:0/3982957096 conn(0x5584846d87b0 msgr2=0x5584846ddbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3982957096 shutdown_connections 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3982957096 wait complete. 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- start start 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 Processor -- start 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dc490 0x5584846ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846efc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ae300 con 0x5584846e7480 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981af770 con 0x5584846dcdb0 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ae9d0 con 0x5584846dc490 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.602+0000 7fc19fa4a740 10 client.0 _ll_register_callbacks cb 0x7fc190005b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43450/0 (socket says 172.21.15.26:43450) 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 -- 172.21.15.26:0/3173968079 learned_addr learned my addr 172.21.15.26:0/3173968079 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc195d9b640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc19659c640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dc490 0x5584846ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 -- 172.21.15.26:0/3173968079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dc490 msgr2=0x5584846ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dc490 0x5584846ef6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 -- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 msgr2=0x5584846efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 -- 172.21.15.26:0/3173968079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584846eaf60 con 0x5584846e7480 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc195d9b640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc196d9d640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc190011010 tx=0x7fc190005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a637cc109c4c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc17dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19003d070 con 0x5584846e7480 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc1900136c0 con 0x5584846e7480 2024-04-23T18:18:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1880a0670 con 0x5584846e7480 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc190046050 con 0x5584846e7480 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc19fa4a740 10 client.15753 fetch_fsmap learned FSMap version 27 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55848475e6c0 con 0x5584846e7480 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.603+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1880a0670 con 0x5584846e7480 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fc190040020 con 0x5584846e7480 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 10 client.15753 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 10 client.15753 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 20 client.15753 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55848475ecc0 0x5584847610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 4 client.15753 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1ea84704", "release": true, "await": true}] 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1ea84704", "release": true, "await": true}) -- 0x55848464f0f0 con 0x55848475ecc0 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19659c640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55848475ecc0 0x5584847610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc17dffb640 10 client.15753 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.604+0000 7fc19659c640 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55848475ecc0 0x5584847610b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ddffcf181307bab server_cookie=6d820c9034070a32 in_seq=0 out_seq=0 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc17dffb640 1 -- 172.21.15.26:0/3173968079 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55848464f0f0 con 0x55848475ecc0 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc17dffb640 10 client.15753 handle_command_reply: tid=0 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 client.15753 shutdown 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 msgr2=0x5584846ea790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc190011010 tx=0x7fc190005e70 comp rx=0 tx=0).stop 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 shutdown_connections 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55848475ecc0 0x5584847610b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5584846dcdb0 0x5584846efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc17dffb640 0 client.15753 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5584846e7480 0x5584846ea790 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 --2- 172.21.15.26:0/3173968079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5584846dc490 0x5584846ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 >> 172.21.15.26:0/3173968079 conn(0x5584846d87b0 msgr2=0x5584846de470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 shutdown_connections 2024-04-23T18:18:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 1 -- 172.21.15.26:0/3173968079 wait complete. 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.607+0000 7fc19fa4a740 20 client.15753 trim_cache size 0 max 16384 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 27, 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 8, 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "1ea84704": { 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 8, 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:18:27.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:27.607 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 msgr2=0x7fc1800a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 0x7fc1800a6330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc190021000 tx=0x7fc19001e000 comp rx=0 tx=0).stop 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 msgr2=0x7fc198102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc18800d770 tx=0x7fc18800dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 shutdown_connections 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc1800a3e70 0x7fc1800a6330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc198106170 0x7fc198103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1981035b0 0x7fc1981a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 --2- 172.21.15.26:0/3174687800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc1981010a0 0x7fc198102b30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 >> 172.21.15.26:0/3174687800 conn(0x7fc198077eb0 msgr2=0x7fc19806a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 shutdown_connections 2024-04-23T18:18:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:27.608+0000 7fc19cd30640 1 -- 172.21.15.26:0/3174687800 wait complete. 2024-04-23T18:18:27.613 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1ea84704', seconds elapsed: 11.2 2024-04-23T18:18:27.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 32365 8.78 MB/sec warmup 22 sec latency 10536.090 ms 2024-04-23T18:18:28.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/755997481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:28.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/755997481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:28.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/755997481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:28.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 35338 9.03 MB/sec warmup 23 sec latency 10.729 ms 2024-04-23T18:18:29.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:28 smithi110 ceph-mon[32585]: pgmap v43: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:28 smithi026 ceph-mon[29364]: pgmap v43: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:28 smithi131 ceph-mon[33041]: pgmap v43: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 781 KiB/s wr, 48 op/s 2024-04-23T18:18:29.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 37843 9.13 MB/sec warmup 24 sec latency 25.009 ms 2024-04-23T18:18:30.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 40045 9.29 MB/sec warmup 25 sec latency 26.759 ms 2024-04-23T18:18:31.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:30 smithi110 ceph-mon[32585]: pgmap v44: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 739 KiB/s wr, 46 op/s 2024-04-23T18:18:31.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:30 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:30 smithi026 ceph-mon[29364]: pgmap v44: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 739 KiB/s wr, 46 op/s 2024-04-23T18:18:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:30 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:31.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:30 smithi131 ceph-mon[33041]: pgmap v44: 97 pgs: 97 active+clean; 21 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 739 KiB/s wr, 46 op/s 2024-04-23T18:18:31.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:30 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:31.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 42910 9.62 MB/sec warmup 26 sec latency 26.048 ms 2024-04-23T18:18:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 -- 172.21.15.26:0/3417981460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8658102b80 msgr2=0x7f8658106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 --2- 172.21.15.26:0/3417981460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8658102b80 0x7f8658106bc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8640009900 tx=0x7f864002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 -- 172.21.15.26:0/3417981460 shutdown_connections 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 --2- 172.21.15.26:0/3417981460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581072a0 0x7f865810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 --2- 172.21.15.26:0/3417981460 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8658102b80 0x7f8658106bc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 --2- 172.21.15.26:0/3417981460 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86581021d0 0x7f86581025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 -- 172.21.15.26:0/3417981460 >> 172.21.15.26:0/3417981460 conn(0x7f86580793e0 msgr2=0x7f86580797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 -- 172.21.15.26:0/3417981460 shutdown_connections 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.901+0000 7f865e8e1640 1 -- 172.21.15.26:0/3417981460 wait complete. 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 Processor -- start 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 -- start start 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581021d0 0x7f8658198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f865819cef0 0x7f865819d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8658112230 con 0x7f86581072a0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f86581120b0 con 0x7f86581021d0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.902+0000 7f865e8e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f86581123b0 con 0x7f865819cef0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38540/0 (socket says 172.21.15.26:38540) 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 -- 172.21.15.26:0/2733072826 learned_addr learned my addr 172.21.15.26:0/2733072826 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f8657fff640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581021d0 0x7f8658198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f865ce7e640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f865819cef0 0x7f865819d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 -- 172.21.15.26:0/2733072826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f865819cef0 msgr2=0x7f865819d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f865819cef0 0x7f865819d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 -- 172.21.15.26:0/2733072826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581021d0 msgr2=0x7f8658198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581021d0 0x7f8658198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 -- 172.21.15.26:0/2733072826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865819d9e0 con 0x7f86581072a0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86577fe640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8640005ab0 tx=0x7f8640003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86557fa640 1 -- 172.21.15.26:0/2733072826 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864003f070 con 0x7f86581072a0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865819dc70 con 0x7f86581072a0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86557fa640 1 -- 172.21.15.26:0/2733072826 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8640031b50 con 0x7f86581072a0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f86557fa640 1 -- 172.21.15.26:0/2733072826 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640035030 con 0x7f86581072a0 2024-04-23T18:18:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.903+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86581a5490 con 0x7f86581072a0 2024-04-23T18:18:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.905+0000 7f86557fa640 1 -- 172.21.15.26:0/2733072826 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f8640042070 con 0x7f86581072a0 2024-04-23T18:18:31.907 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 Processor -- start 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 -- start start 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 0x5631a7239a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7244800 0x5631a7246bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5631a719e970 con 0x5631a7239f40 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5631a713b4a0 con 0x5631a7239620 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86615f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5631a719a930 con 0x5631a7244800 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86557fa640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 0x7f86440a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f8657fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 0x5631a7239a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f8657fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38556/0 (socket says 172.21.15.26:38556) 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f8657fff640 1 -- 172.21.15.26:0/3148495620 learned_addr learned my addr 172.21.15.26:0/3148495620 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 0x7f86440a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86577fe640 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7244800 0x5631a7246bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.908 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f86557fa640 1 -- 172.21.15.26:0/2733072826 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f86400f0ff0 con 0x7f86581072a0 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 -- 172.21.15.26:0/3148495620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7244800 msgr2=0x5631a7246bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7244800 0x5631a7246bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 -- 172.21.15.26:0/3148495620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 msgr2=0x5631a72442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.906+0000 7f865ce7e640 1 -- 172.21.15.26:0/3148495620 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a6eb1210 con 0x5631a7239620 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.907+0000 7f865ce7e640 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 0x5631a7239a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8650014db0 tx=0x7f86500117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665aa92f7ac9e777 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.907+0000 7f865ce7e640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 0x7f86440a63a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8650011cb0 tx=0x7f8650016000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.907+0000 7f864e7fc640 1 -- 172.21.15.26:0/3148495620 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865000cd70 con 0x5631a7239620 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.907+0000 7f864e7fc640 1 -- 172.21.15.26:0/3148495620 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8650007b90 con 0x5631a7239620 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.907+0000 7f864e7fc640 1 -- 172.21.15.26:0/3148495620 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650004290 con 0x5631a7239620 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- 172.21.15.26:0/3148495620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 msgr2=0x5631a7239a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 0x5631a7239a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8650014db0 tx=0x7f86500117b0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- 172.21.15.26:0/3148495620 shutdown_connections 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7244800 0x5631a7246bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.909 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- 172.21.15.26:0/3148495620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239620 0x5631a7239a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- 172.21.15.26:0/3148495620 >> 172.21.15.26:0/3148495620 conn(0x5631a7235b30 msgr2=0x5631a7235f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- 172.21.15.26:0/3148495620 shutdown_connections 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- 172.21.15.26:0/3148495620 wait complete. 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- start start 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 Processor -- start 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a72441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7241ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5631a72473d0 con 0x5631a7239f40 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5631a71390b0 con 0x5631a7244800 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.908+0000 7f86615f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5631a7247550 con 0x5631a7239620 2024-04-23T18:18:31.910 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86615f9740 10 client.0 _ll_register_callbacks cb 0x5631a7126d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38570/0 (socket says 172.21.15.26:38570) 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 -- 172.21.15.26:0/242525935 learned_addr learned my addr 172.21.15.26:0/242525935 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f865ce7e640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7241ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f8657fff640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a72441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 -- 172.21.15.26:0/242525935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 msgr2=0x5631a72441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a72441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 -- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 msgr2=0x5631a7241ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7241ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 -- 172.21.15.26:0/242525935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a6f7d620 con 0x5631a7239f40 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86577fe640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f86400bcbf0 tx=0x7f864002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5998226dbf99a04c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86377fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400bf070 con 0x5631a7239f40 2024-04-23T18:18:31.911 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8650007b90 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f86400b9560 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8640049040 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86615f9740 10 client.15765 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.909+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5631a7253360 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8650007b90 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f86400bd150 con 0x5631a7239f40 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 10 client.15765 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 20 client.15765 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a7254810 0x5631a7256c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 4 client.15765 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631a71ac390 con 0x5631a7254810 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f8657fff640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a7254810 0x5631a7256c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f86377fe640 10 client.15765 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.910+0000 7f8657fff640 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a7254810 0x5631a7256c50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e0eb7399804b9c9 server_cookie=33274369e0d97269 in_seq=0 out_seq=0 2024-04-23T18:18:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.911+0000 7f86377fe640 1 -- 172.21.15.26:0/242525935 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631a71ac390 con 0x5631a7254810 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.911+0000 7f86377fe640 10 client.15765 handle_command_reply: tid=0 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.911+0000 7f86615f9740 1 client.15765 shutdown 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 msgr2=0x5631a72414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f86400bcbf0 tx=0x7f864002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 shutdown_connections 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a7254810 0x5631a7256c50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7241ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a72414b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86377fe640 0 client.15765 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 --2- 172.21.15.26:0/242525935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a72441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 >> 172.21.15.26:0/242525935 conn(0x5631a7235b30 msgr2=0x5631a723b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 shutdown_connections 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 1 -- 172.21.15.26:0/242525935 wait complete. 2024-04-23T18:18:31.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.912+0000 7f86615f9740 20 client.15765 trim_cache size 0 max 16384 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 Processor -- start 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 -- start start 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a7250290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7243cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f86581b02a0 con 0x5631a7239f40 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f86581adab0 con 0x5631a7244800 2024-04-23T18:18:31.928 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86615f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f86581ab900 con 0x5631a7239620 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38586/0 (socket says 172.21.15.26:38586) 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 -- 172.21.15.26:0/978992244 learned_addr learned my addr 172.21.15.26:0/978992244 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f865ce7e640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7243cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f8657fff640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a7250290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 -- 172.21.15.26:0/978992244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 msgr2=0x5631a7250290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a7250290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 -- 172.21.15.26:0/978992244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 msgr2=0x5631a7243cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7243cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 -- 172.21.15.26:0/978992244 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a72572d0 con 0x5631a7239f40 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f8657fff640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a7250290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.929+0000 7f86577fe640 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f86400ee700 tx=0x7f86400bd070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6c5b4bde0afae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f864dffb640 1 -- 172.21.15.26:0/978992244 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400bf050 con 0x5631a7239f40 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f864dffb640 1 -- 172.21.15.26:0/978992244 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86400c5ab0 con 0x5631a7239f40 2024-04-23T18:18:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f864dffb640 1 -- 172.21.15.26:0/978992244 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400c5620 con 0x5631a7239f40 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 -- 172.21.15.26:0/978992244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 msgr2=0x5631a723b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f86400ee700 tx=0x7f86400bd070 comp rx=0 tx=0).stop 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 -- 172.21.15.26:0/978992244 shutdown_connections 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7244800 0x5631a7243cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7239f40 0x5631a723b8b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 --2- 172.21.15.26:0/978992244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a7250290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 -- 172.21.15.26:0/978992244 >> 172.21.15.26:0/978992244 conn(0x5631a7235b30 msgr2=0x5631a723c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 -- 172.21.15.26:0/978992244 shutdown_connections 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.930+0000 7f86615f9740 1 -- 172.21.15.26:0/978992244 wait complete. 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 -- start start 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 Processor -- start 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a723e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 0x5631a723e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f86581abf70 con 0x5631a7244800 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f86581ad3e0 con 0x5631a7239f40 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f86581ac640 con 0x5631a7239620 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86615f9740 10 client.0 _ll_register_callbacks cb 0x7f86400bccc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f865ce7e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f86577fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 0x5631a723e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f865ce7e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38596/0 (socket says 172.21.15.26:38596) 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.931+0000 7f865ce7e640 1 -- 172.21.15.26:0/902859365 learned_addr learned my addr 172.21.15.26:0/902859365 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f8657fff640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a723e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 -- 172.21.15.26:0/902859365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 msgr2=0x5631a723e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a723e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 -- 172.21.15.26:0/902859365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 msgr2=0x5631a723ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 -- 172.21.15.26:0/902859365 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a72b9e50 con 0x5631a7239f40 2024-04-23T18:18:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f865ce7e640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86577fe640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 0x5631a723e7b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x5631a72302d0 tx=0x7f8640003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645f168b2c075461 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86377fe640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864009f020 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f86400bd150 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86400c5ab0 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86400ca050 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86615f9740 10 client.25295 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.932+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5631a72b9e50 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86400c5ab0 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f86400ee8d0 con 0x5631a7239f40 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 10 client.25295 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 10 client.25295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 20 client.25295 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a72bb430 0x5631a72bd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 4 client.25295 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631a71a6200 con 0x5631a72bb430 2024-04-23T18:18:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f8657fff640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a72bb430 0x5631a72bd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f8657fff640 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a72bb430 0x5631a72bd850 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1141f4103cf551a0 server_cookie=b5b7e374208ee31e in_seq=0 out_seq=0 2024-04-23T18:18:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.933+0000 7f86377fe640 10 client.25295 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86377fe640 1 -- 172.21.15.26:0/902859365 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631a71a6200 con 0x5631a72bb430 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86377fe640 10 client.25295 handle_command_reply: tid=0 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 client.25295 shutdown 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 msgr2=0x5631a723e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 0x5631a723e7b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x5631a72302d0 tx=0x7f8640003f90 comp rx=0 tx=0).stop 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 shutdown_connections 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5631a72bb430 0x5631a72bd850 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5631a7239f40 0x5631a723e7b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5631a7244800 0x5631a723ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 --2- 172.21.15.26:0/902859365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631a7239620 0x5631a723e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86377fe640 0 client.25295 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 >> 172.21.15.26:0/902859365 conn(0x5631a7235b30 msgr2=0x5631a723b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 shutdown_connections 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 1 -- 172.21.15.26:0/902859365 wait complete. 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.965+0000 7f86615f9740 20 client.25295 trim_cache size 0 max 16384 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:18:31.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 msgr2=0x7f86440a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 0x7f86440a63a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8650011cb0 tx=0x7f8650016000 comp rx=0 tx=0).stop 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 msgr2=0x7f8658198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8640005ab0 tx=0x7f8640003b90 comp rx=0 tx=0).stop 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 shutdown_connections 2024-04-23T18:18:31.962 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f86440a3ee0 0x7f86440a63a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f865819cef0 0x7f865819d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f86581072a0 0x7f8658198b70 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 --2- 172.21.15.26:0/2733072826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86581021d0 0x7f8658198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 >> 172.21.15.26:0/2733072826 conn(0x7f86580793e0 msgr2=0x7f86580ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 shutdown_connections 2024-04-23T18:18:31.963 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:31.966+0000 7f865e8e1640 1 -- 172.21.15.26:0/2733072826 wait complete. 2024-04-23T18:18:32.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 45475 9.69 MB/sec warmup 27 sec latency 25.880 ms 2024-04-23T18:18:33.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:32 smithi110 ceph-mon[32585]: pgmap v45: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 997 KiB/s wr, 88 op/s 2024-04-23T18:18:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:32 smithi026 ceph-mon[29364]: pgmap v45: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 997 KiB/s wr, 88 op/s 2024-04-23T18:18:33.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:32 smithi131 ceph-mon[33041]: pgmap v45: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 997 KiB/s wr, 88 op/s 2024-04-23T18:18:33.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 47370 9.70 MB/sec warmup 28 sec latency 11.167 ms 2024-04-23T18:18:34.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 50079 9.95 MB/sec warmup 29 sec latency 11.065 ms 2024-04-23T18:18:35.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:34 smithi110 ceph-mon[32585]: pgmap v46: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 347 KiB/s wr, 51 op/s 2024-04-23T18:18:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:34 smithi026 ceph-mon[29364]: pgmap v46: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 347 KiB/s wr, 51 op/s 2024-04-23T18:18:35.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:34 smithi131 ceph-mon[33041]: pgmap v46: 97 pgs: 97 active+clean; 25 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 347 KiB/s wr, 51 op/s 2024-04-23T18:18:35.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 52609 10.04 MB/sec warmup 30 sec latency 17.602 ms 2024-04-23T18:18:36.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 53684 9.93 MB/sec warmup 31 sec latency 172.906 ms 2024-04-23T18:18:37.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:36 smithi110 ceph-mon[32585]: pgmap v47: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 987 KiB/s wr, 169 op/s 2024-04-23T18:18:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:36 smithi026 ceph-mon[29364]: pgmap v47: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 987 KiB/s wr, 169 op/s 2024-04-23T18:18:37.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:36 smithi131 ceph-mon[33041]: pgmap v47: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 987 KiB/s wr, 169 op/s 2024-04-23T18:18:37.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 55605 9.84 MB/sec warmup 32 sec latency 282.714 ms 2024-04-23T18:18:38.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 57389 9.91 MB/sec warmup 33 sec latency 10.469 ms 2024-04-23T18:18:39.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:38 smithi110 ceph-mon[32585]: pgmap v48: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:39.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:38 smithi026 ceph-mon[29364]: pgmap v48: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:39.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:38 smithi131 ceph-mon[33041]: pgmap v48: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:39.657 INFO:tasks.workunit.client.0.smithi026.stdout: 1 60397 10.13 MB/sec warmup 34 sec latency 26.673 ms 2024-04-23T18:18:40.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 61988 10.04 MB/sec warmup 35 sec latency 13.763 ms 2024-04-23T18:18:41.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:40 smithi110 ceph-mon[32585]: pgmap v49: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:41.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:40 smithi026 ceph-mon[29364]: pgmap v49: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:41.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:40 smithi131 ceph-mon[33041]: pgmap v49: 97 pgs: 97 active+clean; 31 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 985 KiB/s wr, 166 op/s 2024-04-23T18:18:41.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 64337 10.14 MB/sec warmup 36 sec latency 9.928 ms 2024-04-23T18:18:42.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 67211 10.26 MB/sec warmup 37 sec latency 15.335 ms 2024-04-23T18:18:43.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:42 smithi110 ceph-mon[32585]: pgmap v50: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 394 op/s 2024-04-23T18:18:43.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:42 smithi110 ceph-mon[32585]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:18:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:42 smithi026 ceph-mon[29364]: pgmap v50: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 394 op/s 2024-04-23T18:18:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:42 smithi026 ceph-mon[29364]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:18:43.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:42 smithi131 ceph-mon[33041]: pgmap v50: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.3 MiB/s wr, 394 op/s 2024-04-23T18:18:43.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:42 smithi131 ceph-mon[33041]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:18:43.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 68845 10.25 MB/sec warmup 38 sec latency 10.875 ms 2024-04-23T18:18:44.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 71337 10.34 MB/sec warmup 39 sec latency 30.950 ms 2024-04-23T18:18:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- 172.21.15.26:0/4262186220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b30106170 msgr2=0x7f2b3010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/4262186220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b30106170 0x7f2b3010dd40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009ca0 tx=0x7f2b2802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- 172.21.15.26:0/4262186220 shutdown_connections 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/4262186220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30101aa0 0x7f2b30105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/4262186220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b30106170 0x7f2b3010dd40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/4262186220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b301010f0 0x7f2b301014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- 172.21.15.26:0/4262186220 >> 172.21.15.26:0/4262186220 conn(0x7f2b30077fd0 msgr2=0x7f2b300783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- 172.21.15.26:0/4262186220 shutdown_connections 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- 172.21.15.26:0/4262186220 wait complete. 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 Processor -- start 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.058+0000 7f2b37bcd640 1 -- start start 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30106170 0x7f2b30198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b3019cf70 0x7f2b3019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30111130 con 0x7f2b301010f0 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30110fb0 con 0x7f2b30106170 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301112b0 con 0x7f2b3019cf70 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57702/0 (socket says 172.21.15.26:57702) 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 -- 172.21.15.26:0/1994251281 learned_addr learned my addr 172.21.15.26:0/1994251281 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35168640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30106170 0x7f2b30198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b3616a640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b3019cf70 0x7f2b3019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 -- 172.21.15.26:0/1994251281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b3019cf70 msgr2=0x7f2b3019d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b3019cf70 0x7f2b3019d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 -- 172.21.15.26:0/1994251281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30106170 msgr2=0x7f2b30198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30106170 0x7f2b30198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.055 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 -- 172.21.15.26:0/1994251281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3019da60 con 0x7f2b301010f0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b35969640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e9f0 tx=0x7f2b2c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b26ffd640 1 -- 172.21.15.26:0/1994251281 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c00cd80 con 0x7f2b301010f0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b26ffd640 1 -- 172.21.15.26:0/1994251281 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b2c0054d0 con 0x7f2b301010f0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3019dd50 con 0x7f2b301010f0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.059+0000 7f2b26ffd640 1 -- 172.21.15.26:0/1994251281 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c010690 con 0x7f2b301010f0 2024-04-23T18:18:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.060+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b301a54f0 con 0x7f2b301010f0 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b26ffd640 1 -- 172.21.15.26:0/1994251281 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2b2c004e90 con 0x7f2b301010f0 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 Processor -- start 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 -- start start 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b2a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.062+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x561a6b1f8970 con 0x561a6b293620 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561a6b1954a0 con 0x561a6b29e800 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x561a6b1f4930 con 0x561a6b293f40 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b26ffd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 0x7f2b1c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57712/0 (socket says 172.21.15.26:57712) 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 -- 172.21.15.26:0/4221051141 learned_addr learned my addr 172.21.15.26:0/4221051141 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b35969640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b35168640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b2a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 0x7f2b1c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b26ffd640 1 -- 172.21.15.26:0/1994251281 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2b2c0d0bd0 con 0x7f2b301010f0 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 -- 172.21.15.26:0/4221051141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 msgr2=0x561a6b29e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 -- 172.21.15.26:0/4221051141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 msgr2=0x561a6b2a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b2a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 -- 172.21.15.26:0/4221051141 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a6af0b210 con 0x561a6b293620 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b35969640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b35168640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b2a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009430 tx=0x7f2b28031550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4c08ceb48050d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b3616a640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 0x7f2b1c0a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b2802aeb0 tx=0x7f2b2803b000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b1bfff640 1 -- 172.21.15.26:0/4221051141 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2804e070 con 0x561a6b293620 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.063+0000 7f2b1bfff640 1 -- 172.21.15.26:0/4221051141 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b28036040 con 0x561a6b293620 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b1bfff640 1 -- 172.21.15.26:0/4221051141 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28004640 con 0x561a6b293620 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/4221051141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 msgr2=0x561a6b293a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009430 tx=0x7f2b28031550 comp rx=0 tx=0).stop 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/4221051141 shutdown_connections 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b2a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/4221051141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b293a00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/4221051141 >> 172.21.15.26:0/4221051141 conn(0x561a6b28fb30 msgr2=0x561a6b28ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/4221051141 shutdown_connections 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.064+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/4221051141 wait complete. 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 -- start start 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 Processor -- start 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 0x561a6b29e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x561a6b2a1250 con 0x561a6b29e800 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561a6b1930b0 con 0x561a6b293620 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x561a6b2a13d0 con 0x561a6b293f40 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3a9c0740 10 client.0 _ll_register_callbacks cb 0x561a6b180d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3616a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b35969640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 0x561a6b29e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3616a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57718/0 (socket says 172.21.15.26:57718) 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b3616a640 1 -- 172.21.15.26:0/183307251 learned_addr learned my addr 172.21.15.26:0/183307251 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.065+0000 7f2b35168640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 -- 172.21.15.26:0/183307251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 msgr2=0x561a6b29b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 -- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 msgr2=0x561a6b29e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 0x561a6b29e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 -- 172.21.15.26:0/183307251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a6afd7620 con 0x561a6b29e800 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b35969640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 0x561a6b29e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3616a640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2b2802a900 tx=0x7f2b28004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcd28d899384dbb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b197fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2805a070 con 0x561a6b29e800 2024-04-23T18:18:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2b28031af0 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b28036040 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b28066050 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3a9c0740 10 client.15783 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a6b2a1250 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.066+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b28036040 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2b280305c0 con 0x561a6b29e800 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 10 client.15783 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 20 client.15783 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b2ae460 0x561a6b2b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 4 client.15783 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a6b206390 con 0x561a6b2ae460 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b35969640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b2ae460 0x561a6b2b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b35969640 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b2ae460 0x561a6b2b0890 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41cef9f9eff4cab server_cookie=5c490b2c0dd72aa in_seq=0 out_seq=0 2024-04-23T18:18:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.067+0000 7f2b197fa640 10 client.15783 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b197fa640 1 -- 172.21.15.26:0/183307251 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a6b206390 con 0x561a6b2ae460 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b197fa640 10 client.15783 handle_command_reply: tid=0 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 client.15783 shutdown 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 msgr2=0x561a6b29bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2b2802a900 tx=0x7f2b28004940 comp rx=0 tx=0).stop 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 shutdown_connections 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b2ae460 0x561a6b2b0890 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b29e800 0x561a6b29bc70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b29b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b197fa640 0 client.15783 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/183307251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b293620 0x561a6b29e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 >> 172.21.15.26:0/183307251 conn(0x561a6b28fb30 msgr2=0x561a6b295860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 shutdown_connections 2024-04-23T18:18:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/183307251 wait complete. 2024-04-23T18:18:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.068+0000 7f2b3a9c0740 20 client.15783 trim_cache size 0 max 16384 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 Processor -- start 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 -- start start 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b295860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b29dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301b0370 con 0x561a6b293620 2024-04-23T18:18:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301adb80 con 0x561a6b29e800 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301ab9d0 con 0x561a6b293f40 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57724/0 (socket says 172.21.15.26:57724) 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b35969640 1 -- 172.21.15.26:0/1877043130 learned_addr learned my addr 172.21.15.26:0/1877043130 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b35168640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b295860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.084+0000 7f2b3616a640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b29dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 -- 172.21.15.26:0/1877043130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 msgr2=0x561a6b295860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b295860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 -- 172.21.15.26:0/1877043130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 msgr2=0x561a6b29dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b29dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 -- 172.21.15.26:0/1877043130 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a6b299ac0 con 0x561a6b293620 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35168640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b295860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:45.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b3616a640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b29dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b35969640 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0ce3b0 tx=0x7f2b2c06f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83975975fc8990a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b1b7fe640 1 -- 172.21.15.26:0/1877043130 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c09d2a0 con 0x561a6b293620 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b1b7fe640 1 -- 172.21.15.26:0/1877043130 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b2c09cba0 con 0x561a6b293620 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.085+0000 7f2b1b7fe640 1 -- 172.21.15.26:0/1877043130 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0ce5d0 con 0x561a6b293620 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/1877043130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 msgr2=0x561a6b2aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0ce3b0 tx=0x7f2b2c06f750 comp rx=0 tx=0).stop 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/1877043130 shutdown_connections 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b29dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b295860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/1877043130 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b2aa110 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/1877043130 >> 172.21.15.26:0/1877043130 conn(0x561a6b28fb30 msgr2=0x561a6b296140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/1877043130 shutdown_connections 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/1877043130 wait complete. 2024-04-23T18:18:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 -- start start 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.086+0000 7f2b3a9c0740 1 Processor -- start 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b2977e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b297d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301ac040 con 0x561a6b293620 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301ad4b0 con 0x561a6b29e800 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301ac710 con 0x561a6b293f40 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 10 client.0 _ll_register_callbacks cb 0x7f2b28032f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57728/0 (socket says 172.21.15.26:57728) 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 -- 172.21.15.26:0/3138698578 learned_addr learned my addr 172.21.15.26:0/3138698578 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35168640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b2977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3616a640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b297d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 -- 172.21.15.26:0/3138698578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 msgr2=0x561a6b2977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b2977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 -- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 msgr2=0x561a6b297d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b297d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 -- 172.21.15.26:0/3138698578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a6b2982e0 con 0x561a6b293620 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b35969640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0097d0 tx=0x7f2b2c0a0fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f018857af3684ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b197fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0b5070 con 0x561a6b293620 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2b2c0ce5d0 con 0x561a6b293620 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.087+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b2c09cba0 con 0x561a6b293620 2024-04-23T18:18:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b2c09d2a0 con 0x561a6b293620 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 10 client.15789 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a6b3197d0 con 0x561a6b293620 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b2c09cba0 con 0x561a6b293620 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2b2c0988f0 con 0x561a6b293620 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 10 client.15789 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 20 client.15789 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b319f60 0x561a6b31c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 4 client.15789 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a6b200200 con 0x561a6b319f60 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.088+0000 7f2b35168640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b319f60 0x561a6b31c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.089+0000 7f2b197fa640 10 client.15789 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.089+0000 7f2b35168640 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b319f60 0x561a6b31c380 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb7d56e1d5b4976e server_cookie=89279f0d6bda3aa2 in_seq=0 out_seq=0 2024-04-23T18:18:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.126+0000 7f2b197fa640 1 -- 172.21.15.26:0/3138698578 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561a6b200200 con 0x561a6b319f60 2024-04-23T18:18:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.126+0000 7f2b197fa640 10 client.15789 handle_command_reply: tid=0 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.126+0000 7f2b3a9c0740 1 client.15789 shutdown 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.126+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 msgr2=0x561a6b298e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.126+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0097d0 tx=0x7f2b2c0a0fc0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 shutdown_connections 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561a6b319f60 0x561a6b31c380 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a6b29e800 0x561a6b297d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561a6b293f40 0x561a6b2977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 --2- 172.21.15.26:0/3138698578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561a6b293620 0x561a6b298e80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b197fa640 0 client.15789 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 >> 172.21.15.26:0/3138698578 conn(0x561a6b28fb30 msgr2=0x561a6b2956e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 shutdown_connections 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 1 -- 172.21.15.26:0/3138698578 wait complete. 2024-04-23T18:18:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b3a9c0740 20 client.15789 trim_cache size 0 max 16384 2024-04-23T18:18:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:18:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:18:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:18:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 msgr2=0x7f2b1c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 0x7f2b1c0a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b2802aeb0 tx=0x7f2b2803b000 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 msgr2=0x7f2b301986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e9f0 tx=0x7f2b2c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 shutdown_connections 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2b1c0a3dc0 0x7f2b1c0a6280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b30106170 0x7f2b30198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2b3019cf70 0x7f2b3019d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 --2- 172.21.15.26:0/1994251281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2b301010f0 0x7f2b301986b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.127+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 >> 172.21.15.26:0/1994251281 conn(0x7f2b30077fd0 msgr2=0x7f2b30069840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.128+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 shutdown_connections 2024-04-23T18:18:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:45.128+0000 7f2b37bcd640 1 -- 172.21.15.26:0/1994251281 wait complete. 2024-04-23T18:18:45.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:44 smithi110 ceph-mon[32585]: pgmap v51: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-04-23T18:18:45.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:44 smithi026 ceph-mon[29364]: pgmap v51: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-04-23T18:18:45.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:44 smithi131 ceph-mon[33041]: pgmap v51: 97 pgs: 97 active+clean; 44 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 346 op/s 2024-04-23T18:18:45.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73598 10.29 MB/sec warmup 40 sec latency 28.895 ms 2024-04-23T18:18:46.186 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-23T18:18:46.186 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 -- 172.21.15.26:0/427952404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e58101ac0 msgr2=0x7f8e58105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 --2- 172.21.15.26:0/427952404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e58101ac0 0x7f8e58105ab0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009900 tx=0x7f8e4802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 -- 172.21.15.26:0/427952404 shutdown_connections 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 --2- 172.21.15.26:0/427952404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58106190 0x7f8e5810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 --2- 172.21.15.26:0/427952404 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e58101ac0 0x7f8e58105ab0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 --2- 172.21.15.26:0/427952404 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58101110 0x7f8e581014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 -- 172.21.15.26:0/427952404 >> 172.21.15.26:0/427952404 conn(0x7f8e58077fb0 msgr2=0x7f8e580783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 -- 172.21.15.26:0/427952404 shutdown_connections 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.304+0000 7f8e605cd640 1 -- 172.21.15.26:0/427952404 wait complete. 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 Processor -- start 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 -- start start 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58101110 0x7f8e5810e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 0x7f8e5810ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8e58112210 con 0x7f8e5810f170 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8e58112090 con 0x7f8e58101110 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e605cd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8e58112390 con 0x7f8e58106190 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e5eb6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e5eb6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57760/0 (socket says 172.21.15.26:57760) 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.305+0000 7f8e5eb6a640 1 -- 172.21.15.26:0/3430279544 learned_addr learned my addr 172.21.15.26:0/3430279544 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5e369640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58101110 0x7f8e5810e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5db68640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 0x7f8e5810ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 -- 172.21.15.26:0/3430279544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 msgr2=0x7f8e5810ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 0x7f8e5810ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 -- 172.21.15.26:0/3430279544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58101110 msgr2=0x7f8e5810e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58101110 0x7f8e5810e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 -- 172.21.15.26:0/3430279544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e581a7040 con 0x7f8e5810f170 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5db68640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 0x7f8e5810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e5eb6a640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000da40 tx=0x7f8e5000df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50017070 con 0x7f8e5810f170 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e50003e20 con 0x7f8e5810f170 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e581a7330 con 0x7f8e5810f170 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50010460 con 0x7f8e5810f170 2024-04-23T18:18:46.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.306+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e581a7830 con 0x7f8e5810f170 2024-04-23T18:18:46.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.307+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e580691c0 con 0x7f8e5810f170 2024-04-23T18:18:46.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.308+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8e50004020 con 0x7f8e5810f170 2024-04-23T18:18:46.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.309+0000 7f8e4f7fe640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 0x7f8e440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.309+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8e500d0400 con 0x7f8e5810f170 2024-04-23T18:18:46.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.309+0000 7f8e5e369640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 0x7f8e440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.309+0000 7f8e5e369640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 0x7f8e440a6210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e540058c0 tx=0x7f8e5400a2d0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.314+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e5009cad0 con 0x7f8e5810f170 2024-04-23T18:18:46.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.540+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e58101ac0 con 0x7f8e5810f170 2024-04-23T18:18:46.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.541+0000 7f8e4f7fe640 1 -- 172.21.15.26:0/3430279544 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11890 (secure 0 0 0) 0x7f8e5009c3d0 con 0x7f8e5810f170 2024-04-23T18:18:46.534 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:18:46.535 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:18:29.409119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:18:46.535 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 28 2024-04-23T18:18:46.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 msgr2=0x7f8e440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 0x7f8e440a6210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e540058c0 tx=0x7f8e5400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 msgr2=0x7f8e581a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000da40 tx=0x7f8e5000df00 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 shutdown_connections 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8e440a3d50 0x7f8e440a6210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8e5810f170 0x7f8e581a6940 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e58106190 0x7f8e5810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 --2- 172.21.15.26:0/3430279544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e58101110 0x7f8e5810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 >> 172.21.15.26:0/3430279544 conn(0x7f8e58077fb0 msgr2=0x7f8e580ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 shutdown_connections 2024-04-23T18:18:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.543+0000 7f8e605cd640 1 -- 172.21.15.26:0/3430279544 wait complete. 2024-04-23T18:18:46.547 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-23T18:18:46.547 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-23T18:18:46.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 10.04 MB/sec warmup 41 sec latency 994.979 ms 2024-04-23T18:18:46.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 -- 172.21.15.26:0/3970785673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8101ac0 msgr2=0x7f88f8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 --2- 172.21.15.26:0/3970785673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8101ac0 0x7f88f8105ab0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009900 tx=0x7f88ec02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:46.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 -- 172.21.15.26:0/3970785673 shutdown_connections 2024-04-23T18:18:46.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 --2- 172.21.15.26:0/3970785673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f8106190 0x7f88f810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 --2- 172.21.15.26:0/3970785673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8101ac0 0x7f88f8105ab0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 --2- 172.21.15.26:0/3970785673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 -- 172.21.15.26:0/3970785673 >> 172.21.15.26:0/3970785673 conn(0x7f88f8077fb0 msgr2=0x7f88f80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 -- 172.21.15.26:0/3970785673 shutdown_connections 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.666+0000 7f88fe89a640 1 -- 172.21.15.26:0/3970785673 wait complete. 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 Processor -- start 2024-04-23T18:18:46.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 -- start start 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f8197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 0x7f88f819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81110d0 con 0x7f88f8106190 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88f8110f50 con 0x7f88f819bf30 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88fe89a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f88f8111250 con 0x7f88f8101110 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.667+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57774/0 (socket says 172.21.15.26:57774) 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 -- 172.21.15.26:0/1031880964 learned_addr learned my addr 172.21.15.26:0/1031880964 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f8197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88fce37640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 0x7f88f819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 -- 172.21.15.26:0/1031880964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 msgr2=0x7f88f8197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f8197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 -- 172.21.15.26:0/1031880964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 msgr2=0x7f88f819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 0x7f88f819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 -- 172.21.15.26:0/1031880964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f819ca20 con 0x7f88f8106190 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88fce37640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 0x7f88f819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88ec032fd0 tx=0x7f88ec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f57fa640 1 -- 172.21.15.26:0/1031880964 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec046070 con 0x7f88f8106190 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f819ccb0 con 0x7f88f8106190 2024-04-23T18:18:46.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f57fa640 1 -- 172.21.15.26:0/1031880964 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ec003aa0 con 0x7f88f8106190 2024-04-23T18:18:46.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88f57fa640 1 -- 172.21.15.26:0/1031880964 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec035050 con 0x7f88f8106190 2024-04-23T18:18:46.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.668+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f806a840 con 0x7f88f8106190 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88f57fa640 1 -- 172.21.15.26:0/1031880964 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f88ec049070 con 0x7f88f8106190 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 Processor -- start 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 -- start start 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b92b110370 con 0x55b92b1aadf0 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b92b0ad2e0 con 0x55b92b1ba4a0 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f89015b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b92b10c330 con 0x55b92b1ab7a0 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88f57fa640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 0x7f88e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88fce37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88fce37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57778/0 (socket says 172.21.15.26:57778) 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88fce37640 1 -- 172.21.15.26:0/2688944262 learned_addr learned my addr 172.21.15.26:0/2688944262 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88f77fe640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.671+0000 7f88fce37640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 0x7f88e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88f57fa640 1 -- 172.21.15.26:0/1031880964 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f88ec0f23b0 con 0x7f88f8106190 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88f7fff640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 -- 172.21.15.26:0/2688944262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 msgr2=0x55b92b1b9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 -- 172.21.15.26:0/2688944262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 msgr2=0x55b92b1bc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 -- 172.21.15.26:0/2688944262 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b92aeeed80 con 0x55b92b1aadf0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88f7fff640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88f0014e50 tx=0x7f88f0011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4c0b29c1b3d2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88fce37640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 0x7f88e00a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88f0011b90 tx=0x7f88f0016000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88de7fc640 1 -- 172.21.15.26:0/2688944262 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0022070 con 0x55b92b1aadf0 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88de7fc640 1 -- 172.21.15.26:0/2688944262 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88f0009cd0 con 0x55b92b1aadf0 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.672+0000 7f88de7fc640 1 -- 172.21.15.26:0/2688944262 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0005280 con 0x55b92b1aadf0 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- 172.21.15.26:0/2688944262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 msgr2=0x55b92b1ab1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88f0014e50 tx=0x7f88f0011750 comp rx=0 tx=0).stop 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- 172.21.15.26:0/2688944262 shutdown_connections 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 --2- 172.21.15.26:0/2688944262 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1ab1d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- 172.21.15.26:0/2688944262 >> 172.21.15.26:0/2688944262 conn(0x55b92b1a73c0 msgr2=0x55b92b1a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- 172.21.15.26:0/2688944262 shutdown_connections 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- 172.21.15.26:0/2688944262 wait complete. 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.673+0000 7f89015b4740 1 -- start start 2024-04-23T18:18:46.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 Processor -- start 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b92b1bd200 con 0x55b92b1aadf0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b92b0aafb0 con 0x55b92b1ba4a0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b92b1bd380 con 0x55b92b1ab7a0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f89015b4740 10 client.0 _ll_register_callbacks cb 0x55b92b0aa9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57780/0 (socket says 172.21.15.26:57780) 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 -- 172.21.15.26:0/1489004562 learned_addr learned my addr 172.21.15.26:0/1489004562 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88fce37640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 -- 172.21.15.26:0/1489004562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 msgr2=0x55b92b1b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 -- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 msgr2=0x55b92b1b7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 -- 172.21.15.26:0/1489004562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b92af4f510 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f88e800bfc0 tx=0x7f88e8011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d7fc666297cf8c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.674+0000 7f88d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8012710 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f88e80128b0 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f0009cd0 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88e8012af0 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 10 client.15807 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b92ade2670 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88f0009cd0 con 0x55b92b1aadf0 2024-04-23T18:18:46.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f88e80233d0 con 0x55b92b1aadf0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 10 client.15807 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 10 client.15807 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 20 client.15807 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b1ca9d0 0x55b92b1cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 4 client.15807 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b92b11dd90 con 0x55b92b1ca9d0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.675+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b1ca9d0 0x55b92b1cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.676+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b1ca9d0 0x55b92b1cce50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8abe93e920c95636 server_cookie=de3ec88adec88da3 in_seq=0 out_seq=0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.676+0000 7f88d77fe640 10 client.15807 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f88d77fe640 1 -- 172.21.15.26:0/1489004562 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b92b11dd90 con 0x55b92b1ca9d0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f88d77fe640 10 client.15807 handle_command_reply: tid=0 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 client.15807 shutdown 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 msgr2=0x55b92b1b7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f88e800bfc0 tx=0x7f88e8011870 comp rx=0 tx=0).stop 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 shutdown_connections 2024-04-23T18:18:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b1ca9d0 0x55b92b1cce50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1b7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ab7a0 0x55b92b1b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f88d77fe640 0 client.15807 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 --2- 172.21.15.26:0/1489004562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1aadf0 0x55b92b1b7490 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 >> 172.21.15.26:0/1489004562 conn(0x55b92b1a73c0 msgr2=0x55b92b1ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 shutdown_connections 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 1 -- 172.21.15.26:0/1489004562 wait complete. 2024-04-23T18:18:46.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.677+0000 7f89015b4740 20 client.15807 trim_cache size 0 max 16384 2024-04-23T18:18:46.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.694+0000 7f89015b4740 1 Processor -- start 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.694+0000 7f89015b4740 1 -- start start 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.694+0000 7f89015b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1aadf0 0x55b92b1c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.694+0000 7f89015b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.694+0000 7f89015b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 0x55b92b1d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f89015b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81ae1c0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f89015b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81ab9d0 con 0x55b92b1aadf0 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f89015b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81a9820 con 0x55b92b1ba4a0 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57784/0 (socket says 172.21.15.26:57784) 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 -- 172.21.15.26:0/662871705 learned_addr learned my addr 172.21.15.26:0/662871705 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88fce37640 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 0x55b92b1d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 -- 172.21.15.26:0/662871705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 msgr2=0x55b92b1d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 0x55b92b1d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 -- 172.21.15.26:0/662871705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1aadf0 msgr2=0x55b92b1c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1aadf0 0x55b92b1c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 -- 172.21.15.26:0/662871705 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b92b1ab540 con 0x55b92b1ab7a0 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88fce37640 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 0x55b92b1d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88f77fe640 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0bad50 tx=0x7f88ec08dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c7142f6ce296be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88ddffb640 1 -- 172.21.15.26:0/662871705 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0be380 con 0x55b92b1ab7a0 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88ddffb640 1 -- 172.21.15.26:0/662871705 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ec0bc680 con 0x55b92b1ab7a0 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.695+0000 7f88ddffb640 1 -- 172.21.15.26:0/662871705 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0efdb0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- 172.21.15.26:0/662871705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 msgr2=0x55b92b1a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0bad50 tx=0x7f88ec08dc60 comp rx=0 tx=0).stop 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- 172.21.15.26:0/662871705 shutdown_connections 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1ba4a0 0x55b92b1d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1a6480 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 --2- 172.21.15.26:0/662871705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1aadf0 0x55b92b1c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- 172.21.15.26:0/662871705 >> 172.21.15.26:0/662871705 conn(0x55b92b1a73c0 msgr2=0x55b92b1a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- 172.21.15.26:0/662871705 shutdown_connections 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- 172.21.15.26:0/662871705 wait complete. 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 -- start start 2024-04-23T18:18:46.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.696+0000 7f89015b4740 1 Processor -- start 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1aadf0 0x55b92b1c2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81a9e90 con 0x55b92b1ab7a0 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81ab300 con 0x55b92b1ba4a0 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81aa560 con 0x55b92b1aadf0 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f89015b4740 10 client.0 _ll_register_callbacks cb 0x7f88e800bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57798/0 (socket says 172.21.15.26:57798) 2024-04-23T18:18:46.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 -- 172.21.15.26:0/1925039100 learned_addr learned my addr 172.21.15.26:0/1925039100 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1aadf0 0x55b92b1c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88fce37640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 -- 172.21.15.26:0/1925039100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1aadf0 msgr2=0x55b92b1c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1aadf0 0x55b92b1c2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 -- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 msgr2=0x55b92b1bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 -- 172.21.15.26:0/1925039100 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b92b1bfe90 con 0x55b92b1ab7a0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88f77fe640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f88ec031440 tx=0x7f88ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35bdc073a27e3c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.697+0000 7f88d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0efdb0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f88ec0ba9f0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88ec0bc680 con 0x55b92b1ab7a0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88ec0be5e0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f89015b4740 10 client.15813 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:46.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b92b231ad0 con 0x55b92b1ab7a0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.698+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ec0bc680 con 0x55b92b1ab7a0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f88ec0f2040 con 0x55b92b1ab7a0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 10 client.15813 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 10 client.15813 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 20 client.15813 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b2320f0 0x55b92b234510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 4 client.15813 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x55b92b117c00 con 0x55b92b2320f0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b2320f0 0x55b92b234510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f88f7fff640 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b2320f0 0x55b92b234510 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63f0110f3c214a17 server_cookie=48e959af7c5a0c9e in_seq=0 out_seq=0 2024-04-23T18:18:46.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.699+0000 7f88d77fe640 10 client.15813 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.775+0000 7f88d77fe640 1 -- 172.21.15.26:0/1925039100 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b92b117c00 con 0x55b92b2320f0 2024-04-23T18:18:46.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.775+0000 7f88d77fe640 10 client.15813 handle_command_reply: tid=0 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.775+0000 7f89015b4740 1 client.15813 shutdown 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 msgr2=0x55b92b1c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f88ec031440 tx=0x7f88ec000bc0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 shutdown_connections 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b92b2320f0 0x55b92b234510 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b92b1ba4a0 0x55b92b1bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b92b1ab7a0 0x55b92b1c2b50 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88d77fe640 0 client.15813 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 --2- 172.21.15.26:0/1925039100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b92b1aadf0 0x55b92b1c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 >> 172.21.15.26:0/1925039100 conn(0x55b92b1a73c0 msgr2=0x55b92b1a6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 shutdown_connections 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 1 -- 172.21.15.26:0/1925039100 wait complete. 2024-04-23T18:18:46.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f89015b4740 20 client.15813 trim_cache size 0 max 16384 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 28, 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 10, 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "5468baf7": { 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 10, 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:46.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:46.772 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 msgr2=0x7f88e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 0x7f88e00a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88f0011b90 tx=0x7f88f0016000 comp rx=0 tx=0).stop 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 msgr2=0x7f88f8197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88ec032fd0 tx=0x7f88ec004270 comp rx=0 tx=0).stop 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 shutdown_connections 2024-04-23T18:18:46.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88e00a3d20 0x7f88e00a61e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88f819bf30 0x7f88f819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f88f8106190 0x7f88f8197bb0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 --2- 172.21.15.26:0/1031880964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f88f8101110 0x7f88f8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.776+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 >> 172.21.15.26:0/1031880964 conn(0x7f88f8077fb0 msgr2=0x7f88f80ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.777+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 shutdown_connections 2024-04-23T18:18:46.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:46.777+0000 7f88fe89a640 1 -- 172.21.15.26:0/1031880964 wait complete. 2024-04-23T18:18:46.781 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5468baf7', quiesce took 0.6 seconds. Will release after: 10 2024-04-23T18:18:47.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:46 smithi026 ceph-mon[29364]: pgmap v52: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.6 MiB/s wr, 478 op/s 2024-04-23T18:18:47.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:46 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3430279544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:47.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:46 smithi131 ceph-mon[33041]: pgmap v52: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.6 MiB/s wr, 478 op/s 2024-04-23T18:18:47.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:46 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3430279544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:47.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:46 smithi110 ceph-mon[32585]: pgmap v52: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.6 MiB/s wr, 478 op/s 2024-04-23T18:18:47.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:46 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3430279544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:47.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 9.80 MB/sec warmup 42 sec latency 1995.070 ms 2024-04-23T18:18:48.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 9.57 MB/sec warmup 43 sec latency 2995.199 ms 2024-04-23T18:18:49.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:48 smithi026 ceph-mon[29364]: pgmap v53: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:49.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:48 smithi131 ceph-mon[33041]: pgmap v53: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:48 smithi110 ceph-mon[32585]: pgmap v53: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:49.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 9.35 MB/sec warmup 44 sec latency 3995.300 ms 2024-04-23T18:18:50.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 9.14 MB/sec warmup 45 sec latency 4995.369 ms 2024-04-23T18:18:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:50 smithi026 ceph-mon[29364]: pgmap v54: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:51.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:50 smithi131 ceph-mon[33041]: pgmap v54: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:51.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:50 smithi110 ceph-mon[32585]: pgmap v54: 97 pgs: 97 active+clean; 49 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.9 MiB/s wr, 360 op/s 2024-04-23T18:18:51.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.95 MB/sec warmup 46 sec latency 5995.481 ms 2024-04-23T18:18:52.658 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.76 MB/sec warmup 47 sec latency 6995.584 ms 2024-04-23T18:18:53.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:52 smithi026 ceph-mon[29364]: pgmap v55: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-23T18:18:53.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:52 smithi131 ceph-mon[33041]: pgmap v55: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-23T18:18:53.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:52 smithi110 ceph-mon[32585]: pgmap v55: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-23T18:18:53.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.57 MB/sec warmup 48 sec latency 7995.706 ms 2024-04-23T18:18:54.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.40 MB/sec warmup 49 sec latency 8995.813 ms 2024-04-23T18:18:55.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:54 smithi026 ceph-mon[29364]: pgmap v56: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 642 KiB/s wr, 146 op/s 2024-04-23T18:18:55.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:54 smithi131 ceph-mon[33041]: pgmap v56: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 642 KiB/s wr, 146 op/s 2024-04-23T18:18:55.244 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:54 smithi110 ceph-mon[32585]: pgmap v56: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 642 KiB/s wr, 146 op/s 2024-04-23T18:18:55.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.23 MB/sec warmup 50 sec latency 9995.946 ms 2024-04-23T18:18:56.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 73603 8.07 MB/sec warmup 51 sec latency 10996.097 ms 2024-04-23T18:18:56.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 -- 172.21.15.26:0/180619299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103b50 msgr2=0x7f889410c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 --2- 172.21.15.26:0/180619299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103b50 0x7f889410c030 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8884009900 tx=0x7f888402a480 comp rx=0 tx=0).stop 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 -- 172.21.15.26:0/180619299 shutdown_connections 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 --2- 172.21.15.26:0/180619299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889410c570 0x7f889410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 --2- 172.21.15.26:0/180619299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103b50 0x7f889410c030 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 --2- 172.21.15.26:0/180619299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8894103230 0x7f8894103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 -- 172.21.15.26:0/180619299 >> 172.21.15.26:0/180619299 conn(0x7f8894077fb0 msgr2=0x7f88940783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:56.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 -- 172.21.15.26:0/180619299 shutdown_connections 2024-04-23T18:18:56.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.903+0000 7f889a679640 1 -- 172.21.15.26:0/180619299 wait complete. 2024-04-23T18:18:56.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 Processor -- start 2024-04-23T18:18:56.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 -- start start 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 0x7f88941013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f889410c570 0x7f8894101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8894112250 con 0x7f8894103230 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88941120d0 con 0x7f8894103b50 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f889a679640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f88941123d0 con 0x7f889410c570 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f8893fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f8893fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48434/0 (socket says 172.21.15.26:48434) 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f8893fff640 1 -- 172.21.15.26:0/3897872117 learned_addr learned my addr 172.21.15.26:0/3897872117 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.904+0000 7f88937fe640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 0x7f88941013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8898c16640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f889410c570 0x7f8894101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:56.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 -- 172.21.15.26:0/3897872117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f889410c570 msgr2=0x7f8894101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f889410c570 0x7f8894101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 -- 172.21.15.26:0/3897872117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 msgr2=0x7f88941013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 0x7f88941013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 -- 172.21.15.26:0/3897872117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88941a6cd0 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f88937fe640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 0x7f88941013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f8893fff640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f887c00d770 tx=0x7f887c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c017070 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f887c0054d0 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88941a6fc0 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c004c30 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.905+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88941a7430 con 0x7f8894103230 2024-04-23T18:18:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.906+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88940691c0 con 0x7f8894103230 2024-04-23T18:18:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.907+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f887c004e10 con 0x7f8894103230 2024-04-23T18:18:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.908+0000 7f88917fa640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 0x7f88800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.908+0000 7f88937fe640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 0x7f88800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.908+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f887c0d0320 con 0x7f8894103230 2024-04-23T18:18:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.908+0000 7f88937fe640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 0x7f88800a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8884031440 tx=0x7f8884005a20 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:56.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:56.913+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f887c09c9f0 con 0x7f8894103230 2024-04-23T18:18:57.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.139+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f889402d5c0 con 0x7f8894103230 2024-04-23T18:18:57.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.140+0000 7f88917fa640 1 -- 172.21.15.26:0/3897872117 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11890 (secure 0 0 0) 0x7f887c09c2f0 con 0x7f8894103230 2024-04-23T18:18:57.133 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:18:57.133 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:18:29.409119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:18:57.134 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 28 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 msgr2=0x7f88800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 0x7f88800a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8884031440 tx=0x7f8884005a20 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 msgr2=0x7f8894100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f887c00d770 tx=0x7f887c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 shutdown_connections 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f88800a3d20 0x7f88800a61e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8894103b50 0x7f88941013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f889410c570 0x7f8894101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 --2- 172.21.15.26:0/3897872117 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8894103230 0x7f8894100e80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 >> 172.21.15.26:0/3897872117 conn(0x7f8894077fb0 msgr2=0x7f88940fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 shutdown_connections 2024-04-23T18:18:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.142+0000 7f889a679640 1 -- 172.21.15.26:0/3897872117 wait complete. 2024-04-23T18:18:57.145 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:56 smithi026 ceph-mon[29364]: pgmap v57: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 KiB/s wr, 151 op/s 2024-04-23T18:18:57.146 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 5468baf7 --release --await' 2024-04-23T18:18:57.147 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 5468baf7 --release --await 2024-04-23T18:18:57.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:56 smithi131 ceph-mon[33041]: pgmap v57: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 KiB/s wr, 151 op/s 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 -- 172.21.15.26:0/1536050955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201021e0 msgr2=0x7f26201025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 --2- 172.21.15.26:0/1536050955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201021e0 0x7f26201025c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2610009900 tx=0x7f261002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 -- 172.21.15.26:0/1536050955 shutdown_connections 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 --2- 172.21.15.26:0/1536050955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2620102b90 0x7f2620106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 --2- 172.21.15.26:0/1536050955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201072b0 0x7f262010ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 --2- 172.21.15.26:0/1536050955 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201021e0 0x7f26201025c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 -- 172.21.15.26:0/1536050955 >> 172.21.15.26:0/1536050955 conn(0x7f26200fda60 msgr2=0x7f26200ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 -- 172.21.15.26:0/1536050955 shutdown_connections 2024-04-23T18:18:57.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.262+0000 7f26267cc640 1 -- 172.21.15.26:0/1536050955 wait complete. 2024-04-23T18:18:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 Processor -- start 2024-04-23T18:18:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 -- start start 2024-04-23T18:18:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26201021e0 0x7f262019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 0x7f2620198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f26201122f0 con 0x7f26201072b0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2620112170 con 0x7f26201021e0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f26267cc640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2620112470 con 0x7f26201987b0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48450/0 (socket says 172.21.15.26:48450) 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f261f7fe640 1 -- 172.21.15.26:0/391462593 learned_addr learned my addr 172.21.15.26:0/391462593 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.263+0000 7f261ffff640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26201021e0 0x7f262019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f2624d69640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 0x7f2620198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 -- 172.21.15.26:0/391462593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 msgr2=0x7f2620198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 0x7f2620198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 -- 172.21.15.26:0/391462593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26201021e0 msgr2=0x7f262019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26201021e0 0x7f262019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 -- 172.21.15.26:0/391462593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26201993b0 con 0x7f26201072b0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f2624d69640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 0x7f2620198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261f7fe640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f261400b450 tx=0x7f261400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261d7fa640 1 -- 172.21.15.26:0/391462593 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2614015020 con 0x7f26201072b0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261d7fa640 1 -- 172.21.15.26:0/391462593 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2614011d40 con 0x7f26201072b0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26201a5440 con 0x7f26201072b0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f261d7fa640 1 -- 172.21.15.26:0/391462593 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261400f340 con 0x7f26201072b0 2024-04-23T18:18:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.264+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26201a5840 con 0x7f26201072b0 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.266+0000 7f261d7fa640 1 -- 172.21.15.26:0/391462593 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2614018020 con 0x7f26201072b0 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.266+0000 7f26295b7740 1 Processor -- start 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 -- start start 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cf00870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b86ce586d0 con 0x55b86cef3490 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b86cdf51a0 con 0x55b86cef3db0 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f26295b7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b86ce54690 con 0x55b86cefe480 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48460/0 (socket says 172.21.15.26:48460) 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 -- 172.21.15.26:0/4060080216 learned_addr learned my addr 172.21.15.26:0/4060080216 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f261ffff640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 -- 172.21.15.26:0/4060080216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 msgr2=0x55b86cf00870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f261d7fa640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 0x7f26080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f261f7fe640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cf00870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cf00870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 -- 172.21.15.26:0/4060080216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 msgr2=0x55b86cefdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f2624d69640 1 -- 172.21.15.26:0/4060080216 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b86cdff5b0 con 0x55b86cef3490 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f261ffff640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.267+0000 7f261d7fa640 1 -- 172.21.15.26:0/391462593 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f26140d0190 con 0x7f26201072b0 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f2624d69640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 0x7f26080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f2624d69640 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2618007f30 tx=0x7f261800c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c7c15fb7b27035 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26067fc640 1 -- 172.21.15.26:0/4060080216 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261800fc90 con 0x55b86cef3490 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26067fc640 1 -- 172.21.15.26:0/4060080216 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2618014070 con 0x55b86cef3490 2024-04-23T18:18:57.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26067fc640 1 -- 172.21.15.26:0/4060080216 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261801bb80 con 0x55b86cef3490 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f2624d69640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 0x7f26080a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f261801b000 tx=0x7f261801e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26295b7740 1 -- 172.21.15.26:0/4060080216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 msgr2=0x55b86cef3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26295b7740 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2618007f30 tx=0x7f261800c750 comp rx=0 tx=0).stop 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.268+0000 7f26295b7740 1 -- 172.21.15.26:0/4060080216 shutdown_connections 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cf00870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- 172.21.15.26:0/4060080216 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3490 0x55b86cef3870 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- 172.21.15.26:0/4060080216 >> 172.21.15.26:0/4060080216 conn(0x55b86ceef7b0 msgr2=0x55b86ceefb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- 172.21.15.26:0/4060080216 shutdown_connections 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- 172.21.15.26:0/4060080216 wait complete. 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- start start 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 Processor -- start 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 0x55b86cefdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cefb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b86cf01310 con 0x55b86cef3db0 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b86cdf2b20 con 0x55b86cef3490 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b86cf01490 con 0x55b86cefe480 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f26295b7740 10 client.0 _ll_register_callbacks cb 0x55b86cdf2530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.269+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48462/0 (socket says 172.21.15.26:48462) 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 -- 172.21.15.26:0/3658276028 learned_addr learned my addr 172.21.15.26:0/3658276028 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261ffff640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 0x55b86cefdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 -- 172.21.15.26:0/3658276028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 msgr2=0x55b86cefb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cefb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 -- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 msgr2=0x55b86cefdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 0x55b86cefdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 -- 172.21.15.26:0/3658276028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b86cb6a920 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261ffff640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 0x55b86cefdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2614009fd0 tx=0x7f261409c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c3e3429bede4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f25ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26140a0360 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2614011890 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2618014070 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f261409c970 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f26295b7740 10 client.15831 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b86cc36d30 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.270+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2618014070 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2614097fc0 con 0x55b86cef3db0 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 10 client.15831 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 10 client.15831 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 20 client.15831 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:57.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf0e5e0 0x55b86cf10a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 4 client.15831 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b86ce16500 con 0x55b86cf0e5e0 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f261ffff640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf0e5e0 0x55b86cf10a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f261ffff640 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf0e5e0 0x55b86cf10a80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd0c2d54e0ba68ad server_cookie=bb0a6cc8e6b11a26 in_seq=0 out_seq=0 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.271+0000 7f25ff7fe640 10 client.15831 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3658276028 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b86ce16500 con 0x55b86cf0e5e0 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f25ff7fe640 10 client.15831 handle_command_reply: tid=0 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 client.15831 shutdown 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 msgr2=0x55b86cefb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2614009fd0 tx=0x7f261409c040 comp rx=0 tx=0).stop 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 shutdown_connections 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf0e5e0 0x55b86cf10a80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cefe480 0x55b86cefb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cefb1a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f25ff7fe640 0 client.15831 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 --2- 172.21.15.26:0/3658276028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3490 0x55b86cefdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 >> 172.21.15.26:0/3658276028 conn(0x55b86ceef7b0 msgr2=0x55b86ceee9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 shutdown_connections 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 1 -- 172.21.15.26:0/3658276028 wait complete. 2024-04-23T18:18:57.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.272+0000 7f26295b7740 20 client.15831 trim_cache size 0 max 16384 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 Processor -- start 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 -- start start 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf0a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f26201b0600 con 0x55b86cefe480 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f26201ade10 con 0x55b86cef3db0 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f26295b7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f26201abc60 con 0x55b86cef3490 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48468/0 (socket says 172.21.15.26:48468) 2024-04-23T18:18:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 -- 172.21.15.26:0/1654558452 learned_addr learned my addr 172.21.15.26:0/1654558452 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f261ffff640 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf0a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f261f7fe640 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 -- 172.21.15.26:0/1654558452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 msgr2=0x55b86cf0a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf0a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 -- 172.21.15.26:0/1654558452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 msgr2=0x55b86cefd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 -- 172.21.15.26:0/1654558452 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b86cef2aa0 con 0x55b86cefe480 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2624d69640 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2618004a20 tx=0x7f261800ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff52269f5786a0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.290+0000 7f2605ffb640 1 -- 172.21.15.26:0/1654558452 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618007b20 con 0x55b86cefe480 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f2605ffb640 1 -- 172.21.15.26:0/1654558452 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2618028070 con 0x55b86cefe480 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f2605ffb640 1 -- 172.21.15.26:0/1654558452 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261801b7f0 con 0x55b86cefe480 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- 172.21.15.26:0/1654558452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 msgr2=0x55b86cefdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2618004a20 tx=0x7f261800ff90 comp rx=0 tx=0).stop 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- 172.21.15.26:0/1654558452 shutdown_connections 2024-04-23T18:18:57.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cef3db0 0x55b86cefd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cefe480 0x55b86cefdf20 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 --2- 172.21.15.26:0/1654558452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf0a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- 172.21.15.26:0/1654558452 >> 172.21.15.26:0/1654558452 conn(0x55b86ceef7b0 msgr2=0x55b86cef54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- 172.21.15.26:0/1654558452 shutdown_connections 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- 172.21.15.26:0/1654558452 wait complete. 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.291+0000 7f26295b7740 1 -- start start 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 Processor -- start 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cefe480 0x55b86cf016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f26201ac2d0 con 0x55b86cef3db0 2024-04-23T18:18:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f26201ad740 con 0x55b86cefe480 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f26201ac9a0 con 0x55b86cef3490 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f26295b7740 10 client.0 _ll_register_callbacks cb 0x7f26140988e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:48484/0 (socket says 172.21.15.26:48484) 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 -- 172.21.15.26:0/3656659514 learned_addr learned my addr 172.21.15.26:0/3656659514 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261ffff640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 -- 172.21.15.26:0/3656659514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 msgr2=0x55b86cf066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf066f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 -- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cefe480 msgr2=0x55b86cf016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cefe480 0x55b86cf016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 -- 172.21.15.26:0/3656659514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b86cf01ef0 con 0x55b86cef3db0 2024-04-23T18:18:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f261f7fe640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f26140cdf10 tx=0x7f2614009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9a10f7653b01d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.292+0000 7f25ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261409bdb0 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f261409b6c0 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2618028070 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f261409c270 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 10 client.15837 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b86cf757a0 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2618028070 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2614064060 con 0x55b86cef3db0 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 10 client.15837 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 10 client.15837 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 20 client.15837 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf75f80 0x55b86cf783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 4 client.15837 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5468baf7", "release": true, "await": true}] 2024-04-23T18:18:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5468baf7", "release": true, "await": true}) -- 0x55b86ce660f0 con 0x55b86cf75f80 2024-04-23T18:18:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.293+0000 7f261ffff640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf75f80 0x55b86cf783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.294+0000 7f25ff7fe640 10 client.15837 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.294+0000 7f261ffff640 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf75f80 0x55b86cf783a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c59c011f73aecccb server_cookie=40cf2cd83fd6c783 in_seq=0 out_seq=0 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f25ff7fe640 1 -- 172.21.15.26:0/3656659514 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b86ce660f0 con 0x55b86cf75f80 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f25ff7fe640 10 client.15837 handle_command_reply: tid=0 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 client.15837 shutdown 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 msgr2=0x55b86cf06c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f26140cdf10 tx=0x7f2614009320 comp rx=0 tx=0).stop 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 shutdown_connections 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b86cf75f80 0x55b86cf783a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b86cefe480 0x55b86cf016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b86cef3db0 0x55b86cf06c30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f25ff7fe640 0 client.15837 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 --2- 172.21.15.26:0/3656659514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b86cef3490 0x55b86cf066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 >> 172.21.15.26:0/3656659514 conn(0x55b86ceef7b0 msgr2=0x55b86cef54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 shutdown_connections 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 1 -- 172.21.15.26:0/3656659514 wait complete. 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26295b7740 20 client.15837 trim_cache size 0 max 16384 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 28, 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 12, 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "5468baf7": { 2024-04-23T18:18:57.295 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 12, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:57.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 msgr2=0x7f26080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 0x7f26080a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f261801b000 tx=0x7f261801e000 comp rx=0 tx=0).stop 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 msgr2=0x7f262019ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.299+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f261400b450 tx=0x7f261400b910 comp rx=0 tx=0).stop 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 shutdown_connections 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f26080a3d20 0x7f26080a61e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26201987b0 0x7f2620198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f26201072b0 0x7f262019ebe0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 --2- 172.21.15.26:0/391462593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26201021e0 0x7f262019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 >> 172.21.15.26:0/391462593 conn(0x7f26200fda60 msgr2=0x7f26200fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 shutdown_connections 2024-04-23T18:18:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:18:57.300+0000 7f26267cc640 1 -- 172.21.15.26:0/391462593 wait complete. 2024-04-23T18:18:57.304 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5468baf7', seconds elapsed: 11.1 2024-04-23T18:18:57.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:56 smithi110 ceph-mon[32585]: pgmap v57: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 KiB/s wr, 151 op/s 2024-04-23T18:18:57.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 74699 8.10 MB/sec warmup 52 sec latency 11639.533 ms 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 -- 172.21.15.26:0/2359060921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701010e0 msgr2=0x7fc5701014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 --2- 172.21.15.26:0/2359060921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701010e0 0x7fc5701014c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc560009900 tx=0x7fc56002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 -- 172.21.15.26:0/2359060921 shutdown_connections 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 --2- 172.21.15.26:0/2359060921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701061b0 0x7fc57010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 --2- 172.21.15.26:0/2359060921 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc570101a90 0x7fc570105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 --2- 172.21.15.26:0/2359060921 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701010e0 0x7fc5701014c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 -- 172.21.15.26:0/2359060921 >> 172.21.15.26:0/2359060921 conn(0x7fc5700771d0 msgr2=0x7fc5700775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 -- 172.21.15.26:0/2359060921 shutdown_connections 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.219+0000 7fc576220640 1 -- 172.21.15.26:0/2359060921 wait complete. 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 Processor -- start 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 -- start start 2024-04-23T18:18:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 0x7fc570199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc57019a030 0x7fc5701a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc570111110 con 0x7fc5701061b0 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc570110f90 con 0x7fc5701010e0 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc576220640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc570111290 con 0x7fc57019a030 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51248/0 (socket says 172.21.15.26:51248) 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc56effd640 1 -- 172.21.15.26:0/1459370 learned_addr learned my addr 172.21.15.26:0/1459370 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc56f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 0x7fc570199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.220+0000 7fc56ffff640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc57019a030 0x7fc5701a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 -- 172.21.15.26:0/1459370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc57019a030 msgr2=0x7fc5701a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc57019a030 0x7fc5701a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 -- 172.21.15.26:0/1459370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 msgr2=0x7fc570199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 0x7fc570199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 -- 172.21.15.26:0/1459370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5701a4db0 con 0x7fc5701061b0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 0x7fc570199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56effd640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e9f0 tx=0x7fc56400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56cff9640 1 -- 172.21.15.26:0/1459370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56400ccc0 con 0x7fc5701061b0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56cff9640 1 -- 172.21.15.26:0/1459370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5640054d0 con 0x7fc5701061b0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5701a50a0 con 0x7fc5701061b0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc56cff9640 1 -- 172.21.15.26:0/1459370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564010430 con 0x7fc5701061b0 2024-04-23T18:18:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.221+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5701a55a0 con 0x7fc5701061b0 2024-04-23T18:18:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.223+0000 7fc56cff9640 1 -- 172.21.15.26:0/1459370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc5640046c0 con 0x7fc5701061b0 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.223+0000 7fc57900f740 1 Processor -- start 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.223+0000 7fc57900f740 1 -- start start 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 0x5597f80332c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5597f7f8d970 con 0x5597f8033800 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5597f7f2a4a0 con 0x5597f8028620 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc57900f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5597f7f89930 con 0x5597f8028f40 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56cff9640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 0x7fc5580a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:52376/0 (socket says 172.21.15.26:52376) 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56f7fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 0x5597f80332c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56ffff640 1 -- 172.21.15.26:0/3217423445 learned_addr learned my addr 172.21.15.26:0/3217423445 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51254/0 (socket says 172.21.15.26:51254) 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56cff9640 1 -- 172.21.15.26:0/1459370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc5640d0b80 con 0x7fc5701061b0 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56ffff640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 0x7fc5580a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 -- 172.21.15.26:0/3217423445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 msgr2=0x5597f80332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 0x5597f80332c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 -- 172.21.15.26:0/3217423445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 msgr2=0x5597f8028a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56effd640 1 -- 172.21.15.26:0/3217423445 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597f7ca0210 con 0x5597f8033800 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56ffff640 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.224+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 0x5597f80332c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc56effd640 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc5640ce2a0 tx=0x7fc564021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45405251eef3873 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc555ffb640 1 -- 172.21.15.26:0/3217423445 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56409d1f0 con 0x5597f8033800 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc56ffff640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 0x7fc5580a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5680049d0 tx=0x7fc568004ab0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc555ffb640 1 -- 172.21.15.26:0/3217423445 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc56409caf0 con 0x5597f8033800 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc555ffb640 1 -- 172.21.15.26:0/3217423445 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5640992d0 con 0x5597f8033800 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 -- 172.21.15.26:0/3217423445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 msgr2=0x5597f8035bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc5640ce2a0 tx=0x7fc564021000 comp rx=0 tx=0).stop 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 -- 172.21.15.26:0/3217423445 shutdown_connections 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8033800 0x5597f8035bf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028f40 0x5597f80332c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 --2- 172.21.15.26:0/3217423445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f8028a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.225+0000 7fc57900f740 1 -- 172.21.15.26:0/3217423445 >> 172.21.15.26:0/3217423445 conn(0x5597f8024b30 msgr2=0x5597f8024f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- 172.21.15.26:0/3217423445 shutdown_connections 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- 172.21.15.26:0/3217423445 wait complete. 2024-04-23T18:18:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- start start 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 Processor -- start 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028620 0x5597f8033150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8033800 0x5597f80309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5597f8036430 con 0x5597f8028f40 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5597f7f280b0 con 0x5597f8033800 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.226+0000 7fc57900f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5597f80365b0 con 0x5597f8028620 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc57900f740 10 client.0 _ll_register_callbacks cb 0x5597f7f15d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51256/0 (socket says 172.21.15.26:51256) 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 -- 172.21.15.26:0/1042431998 learned_addr learned my addr 172.21.15.26:0/1042431998 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028620 0x5597f8033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56ffff640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8033800 0x5597f80309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 -- 172.21.15.26:0/1042431998 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028620 msgr2=0x5597f8033150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028620 0x5597f8033150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 -- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8033800 msgr2=0x5597f80309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8033800 0x5597f80309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 -- 172.21.15.26:0/1042431998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597f7d6c620 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc56effd640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc564098f50 tx=0x7fc564098d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e888b0dbaa07329b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc54effd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5640a54d0 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc564005000 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc56409caf0 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc5640a5760 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc57900f740 10 client.15849 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.227+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5597f8042360 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc56409caf0 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc564098940 con 0x5597f8028f40 2024-04-23T18:18:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 10 client.15849 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 10 client.15849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 20 client.15849 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80436b0 0x5597f8045aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 4 client.15849 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597f7f9b390 con 0x5597f80436b0 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80436b0 0x5597f8045aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc54effd640 10 client.15849 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.228+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80436b0 0x5597f8045aa0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34f0876ce2d4a61c server_cookie=a28dde1f15aec5e4 in_seq=0 out_seq=0 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc54effd640 1 -- 172.21.15.26:0/1042431998 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597f7f9b390 con 0x5597f80436b0 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc54effd640 10 client.15849 handle_command_reply: tid=0 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 client.15849 shutdown 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 msgr2=0x5597f8030440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc564098f50 tx=0x7fc564098d90 comp rx=0 tx=0).stop 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 shutdown_connections 2024-04-23T18:18:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80436b0 0x5597f8045aa0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc54effd640 0 client.15849 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8033800 0x5597f80309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8030440 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 --2- 172.21.15.26:0/1042431998 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8028620 0x5597f8033150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 >> 172.21.15.26:0/1042431998 conn(0x5597f8024b30 msgr2=0x5597f802a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 shutdown_connections 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 1 -- 172.21.15.26:0/1042431998 wait complete. 2024-04-23T18:18:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.230+0000 7fc57900f740 20 client.15849 trim_cache size 0 max 16384 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 Processor -- start 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 -- start start 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8032c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701b0410 con 0x5597f8028f40 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701adc20 con 0x5597f8028620 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc57900f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701aba90 con 0x5597f8033800 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.245+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51262/0 (socket says 172.21.15.26:51262) 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 -- 172.21.15.26:0/389919254 learned_addr learned my addr 172.21.15.26:0/389919254 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56ffff640 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8032c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 -- 172.21.15.26:0/389919254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 msgr2=0x5597f8032c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8032c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 -- 172.21.15.26:0/389919254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 msgr2=0x5597f803f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 -- 172.21.15.26:0/389919254 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597f802f450 con 0x5597f8028f40 2024-04-23T18:18:58.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc56effd640 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc5640a5920 tx=0x7fc56409b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59a244f7f2688a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc5557fa640 1 -- 172.21.15.26:0/389919254 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56409caf0 con 0x5597f8028f40 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc5557fa640 1 -- 172.21.15.26:0/389919254 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5640a0ea0 con 0x5597f8028f40 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.246+0000 7fc5557fa640 1 -- 172.21.15.26:0/389919254 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5640a5530 con 0x5597f8028f40 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- 172.21.15.26:0/389919254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 msgr2=0x5597f8024f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc5640a5920 tx=0x7fc56409b000 comp rx=0 tx=0).stop 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- 172.21.15.26:0/389919254 shutdown_connections 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8032c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8024f10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 --2- 172.21.15.26:0/389919254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- 172.21.15.26:0/389919254 >> 172.21.15.26:0/389919254 conn(0x5597f8024b30 msgr2=0x5597f802baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- 172.21.15.26:0/389919254 shutdown_connections 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- 172.21.15.26:0/389919254 wait complete. 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 -- start start 2024-04-23T18:18:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.247+0000 7fc57900f740 1 Processor -- start 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8037470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701ac0e0 con 0x5597f8028f40 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701ad550 con 0x5597f8028620 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc5701ac7b0 con 0x5597f8033800 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 10 client.0 _ll_register_callbacks cb 0x7fc56400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8037470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51274/0 (socket says 172.21.15.26:51274) 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 -- 172.21.15.26:0/1655045015 learned_addr learned my addr 172.21.15.26:0/1655045015 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:18:58.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 -- 172.21.15.26:0/1655045015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 msgr2=0x5597f8037470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8037470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 -- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 msgr2=0x5597f803bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 -- 172.21.15.26:0/1655045015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597f8028d70 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc56effd640 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc56409c7f0 tx=0x7fc5640bd280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669ba3eccde1195b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc54effd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56409caf0 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc56409baa0 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5640a0ea0 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc56409bd20 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 10 client.15855 fetch_fsmap learned FSMap version 28 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.248+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5597f8028d70 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5640a0ea0 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc5640870d0 con 0x5597f8028f40 2024-04-23T18:18:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 10 client.15855 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 10 client.15855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 20 client.15855 populate_metadata read hostname 'smithi026' 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80a9b30 0x5597f80abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 4 client.15855 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5597f7f95200 con 0x5597f80a9b30 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80a9b30 0x5597f80abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc54effd640 10 client.15855 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.249+0000 7fc56f7fe640 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80a9b30 0x5597f80abf50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fb259b74d7b836 server_cookie=81597897ac94b284 in_seq=0 out_seq=0 2024-04-23T18:18:58.248 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:57 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3897872117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc54effd640 1 -- 172.21.15.26:0/1655045015 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5597f7f95200 con 0x5597f80a9b30 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc54effd640 10 client.15855 handle_command_reply: tid=0 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 client.15855 shutdown 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 msgr2=0x5597f8036f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc56409c7f0 tx=0x7fc5640bd280 comp rx=0 tx=0).stop 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 shutdown_connections 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597f80a9b30 0x5597f80abf50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597f8033800 0x5597f8037470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597f8028f40 0x5597f8036f30 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 --2- 172.21.15.26:0/1655045015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597f8028620 0x5597f803bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc54effd640 0 client.15855 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 >> 172.21.15.26:0/1655045015 conn(0x5597f8024b30 msgr2=0x5597f8023cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.271+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 shutdown_connections 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc57900f740 1 -- 172.21.15.26:0/1655045015 wait complete. 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc57900f740 20 client.15855 trim_cache size 0 max 16384 2024-04-23T18:18:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:18:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:18:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:18:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:18:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 msgr2=0x7fc5580a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 0x7fc5580a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5680049d0 tx=0x7fc568004ab0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 msgr2=0x7fc570199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e9f0 tx=0x7fc56400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 shutdown_connections 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc5580a3dc0 0x7fc5580a6280 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc57019a030 0x7fc5701a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc5701061b0 0x7fc570199a50 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 --2- 172.21.15.26:0/1459370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701010e0 0x7fc570199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 >> 172.21.15.26:0/1459370 conn(0x7fc5700771d0 msgr2=0x7fc5700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 shutdown_connections 2024-04-23T18:18:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:18:58.272+0000 7fc576220640 1 -- 172.21.15.26:0/1459370 wait complete. 2024-04-23T18:18:58.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:57 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3897872117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:58.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:57 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3897872117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:58.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 76192 8.07 MB/sec warmup 53 sec latency 27.334 ms 2024-04-23T18:18:59.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:18:58 smithi131 ceph-mon[33041]: pgmap v58: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:18:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:18:58 smithi110 ceph-mon[32585]: pgmap v58: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:18:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:18:58 smithi026 ceph-mon[29364]: pgmap v58: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:18:59.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 78521 8.18 MB/sec warmup 54 sec latency 10.508 ms 2024-04-23T18:19:00.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 79772 8.13 MB/sec warmup 55 sec latency 20.502 ms 2024-04-23T18:19:01.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:00 smithi131 ceph-mon[33041]: pgmap v59: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:19:01.260 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:00 smithi026 ceph-mon[29364]: pgmap v59: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:19:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:00 smithi110 ceph-mon[32585]: pgmap v59: 97 pgs: 97 active+clean; 49 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 58 KiB/s wr, 18 op/s 2024-04-23T18:19:01.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 82068 8.23 MB/sec warmup 56 sec latency 13.807 ms 2024-04-23T18:19:02.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 84396 8.28 MB/sec warmup 57 sec latency 14.786 ms 2024-04-23T18:19:03.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:02 smithi131 ceph-mon[33041]: pgmap v60: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 479 KiB/s wr, 121 op/s 2024-04-23T18:19:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:02 smithi110 ceph-mon[32585]: pgmap v60: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 479 KiB/s wr, 121 op/s 2024-04-23T18:19:03.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:02 smithi026 ceph-mon[29364]: pgmap v60: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 479 KiB/s wr, 121 op/s 2024-04-23T18:19:03.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 86063 8.29 MB/sec warmup 58 sec latency 13.329 ms 2024-04-23T18:19:04.659 INFO:tasks.workunit.client.0.smithi026.stdout: 1 88942 8.45 MB/sec warmup 59 sec latency 15.263 ms 2024-04-23T18:19:05.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:04 smithi131 ceph-mon[33041]: pgmap v61: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 427 KiB/s wr, 106 op/s 2024-04-23T18:19:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:04 smithi110 ceph-mon[32585]: pgmap v61: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 427 KiB/s wr, 106 op/s 2024-04-23T18:19:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:04 smithi026 ceph-mon[29364]: pgmap v61: 97 pgs: 97 active+clean; 53 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 427 KiB/s wr, 106 op/s 2024-04-23T18:19:05.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 90375 8.41 MB/sec warmup 60 sec latency 11.056 ms 2024-04-23T18:19:06.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 92696 8.49 MB/sec warmup 61 sec latency 12.505 ms 2024-04-23T18:19:07.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:06 smithi131 ceph-mon[33041]: pgmap v62: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 215 op/s 2024-04-23T18:19:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:06 smithi110 ceph-mon[32585]: pgmap v62: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 215 op/s 2024-04-23T18:19:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:06 smithi026 ceph-mon[29364]: pgmap v62: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 215 op/s 2024-04-23T18:19:07.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 94878 8.49 MB/sec warmup 62 sec latency 22.309 ms 2024-04-23T18:19:08.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 96538 8.53 MB/sec warmup 63 sec latency 16.131 ms 2024-04-23T18:19:09.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:08 smithi131 ceph-mon[33041]: pgmap v63: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:08 smithi110 ceph-mon[32585]: pgmap v63: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:08 smithi026 ceph-mon[29364]: pgmap v63: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:09.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 99368 8.67 MB/sec warmup 64 sec latency 13.983 ms 2024-04-23T18:19:10.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 100762 8.64 MB/sec warmup 65 sec latency 11.075 ms 2024-04-23T18:19:11.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:10 smithi131 ceph-mon[33041]: pgmap v64: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 -- 172.21.15.26:0/3932069484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c06a780 msgr2=0x7f6f0c06abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 --2- 172.21.15.26:0/3932069484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c06a780 0x7f6f0c06abe0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009900 tx=0x7f6efc02a480 comp rx=0 tx=0).stop 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 -- 172.21.15.26:0/3932069484 shutdown_connections 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 --2- 172.21.15.26:0/3932069484 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 0x7f6f0c10edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 --2- 172.21.15.26:0/3932069484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c06a780 0x7f6f0c06abe0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 --2- 172.21.15.26:0/3932069484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 -- 172.21.15.26:0/3932069484 >> 172.21.15.26:0/3932069484 conn(0x7f6f0c076c50 msgr2=0x7f6f0c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 -- 172.21.15.26:0/3932069484 shutdown_connections 2024-04-23T18:19:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.364+0000 7f6f12d87640 1 -- 172.21.15.26:0/3932069484 wait complete. 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 Processor -- start 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 -- start start 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 0x7f6f0c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 0x7f6f0c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1121b0 con 0x7f6f0c1080e0 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c112030 con 0x7f6f0c06a780 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f12d87640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c112330 con 0x7f6f0c06b120 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f11324640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.365+0000 7f6f11324640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50052/0 (socket says 172.21.15.26:50052) 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 -- 172.21.15.26:0/84652744 learned_addr learned my addr 172.21.15.26:0/84652744 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f10b23640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 0x7f6f0c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f03fff640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 0x7f6f0c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 -- 172.21.15.26:0/84652744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 msgr2=0x7f6f0c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 0x7f6f0c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 -- 172.21.15.26:0/84652744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 msgr2=0x7f6f0c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 0x7f6f0c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 -- 172.21.15.26:0/84652744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c19d5f0 con 0x7f6f0c1080e0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f10b23640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 0x7f6f0c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f11324640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400eb10 tx=0x7f6f0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f01ffb640 1 -- 172.21.15.26:0/84652744 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04004b20 con 0x7f6f0c1080e0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f01ffb640 1 -- 172.21.15.26:0/84652744 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f04004cc0 con 0x7f6f0c1080e0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f01ffb640 1 -- 172.21.15.26:0/84652744 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04010670 con 0x7f6f0c1080e0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c19d8e0 con 0x7f6f0c1080e0 2024-04-23T18:19:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.366+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c1a5170 con 0x7f6f0c1080e0 2024-04-23T18:19:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.368+0000 7f6f01ffb640 1 -- 172.21.15.26:0/84652744 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6f04005020 con 0x7f6f0c1080e0 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 Processor -- start 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 -- start start 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80620 0x560f29c80a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c8dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560f29be5970 con 0x560f29c8b800 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560f29b824a0 con 0x560f29c80f40 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f15b79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560f29be1930 con 0x560f29c80620 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c8dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35260/0 (socket says 172.21.15.26:35260) 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 -- 172.21.15.26:0/1916486124 learned_addr learned my addr 172.21.15.26:0/1916486124 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f03fff640 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80620 0x560f29c80a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f01ffb640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 0x7f6ef40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 -- 172.21.15.26:0/1916486124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80620 msgr2=0x560f29c80a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80620 0x560f29c80a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 -- 172.21.15.26:0/1916486124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 msgr2=0x560f29c8dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c8dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f10b23640 1 -- 172.21.15.26:0/1916486124 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f298f8210 con 0x560f29c80f40 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.369+0000 7f6f11324640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 0x7f6ef40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6f01ffb640 1 -- 172.21.15.26:0/84652744 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6f040d0d00 con 0x7f6f0c1080e0 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6f10b23640 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009f60 tx=0x7f6f08009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b485473a6218c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6ef2ffd640 1 -- 172.21.15.26:0/1916486124 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08023070 con 0x560f29c80f40 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6ef2ffd640 1 -- 172.21.15.26:0/1916486124 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0800fc70 con 0x560f29c80f40 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6f11324640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 0x7f6ef40a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f0409c9d0 tx=0x7f6f0406b680 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.370+0000 7f6ef2ffd640 1 -- 172.21.15.26:0/1916486124 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08016cc0 con 0x560f29c80f40 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- 172.21.15.26:0/1916486124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 msgr2=0x560f29c8b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009f60 tx=0x7f6f08009f90 comp rx=0 tx=0).stop 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- 172.21.15.26:0/1916486124 shutdown_connections 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c8b2c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c8dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1916486124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80620 0x560f29c80a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- 172.21.15.26:0/1916486124 >> 172.21.15.26:0/1916486124 conn(0x560f29c7cb30 msgr2=0x560f29c7cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- 172.21.15.26:0/1916486124 shutdown_connections 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- 172.21.15.26:0/1916486124 wait complete. 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 -- start start 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.371+0000 7f6f15b79740 1 Processor -- start 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c8b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80f40 0x560f29c88470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560f29c8e430 con 0x560f29c8b800 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560f29b800b0 con 0x560f29c80620 2024-04-23T18:19:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560f29c8e5b0 con 0x560f29c80f40 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 10 client.0 _ll_register_callbacks cb 0x560f29b6dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50064/0 (socket says 172.21.15.26:50064) 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 -- 172.21.15.26:0/3546090755 learned_addr learned my addr 172.21.15.26:0/3546090755 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f10b23640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80f40 0x560f29c88470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c8b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 -- 172.21.15.26:0/3546090755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80f40 msgr2=0x560f29c88470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80f40 0x560f29c88470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 -- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 msgr2=0x560f29c8b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c8b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 -- 172.21.15.26:0/3546090755 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f299c4620 con 0x560f29c8b800 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f03fff640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6efc005c40 tx=0x7f6efc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47246283aeef1415 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6eebfff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc035040 con 0x560f29c8b800 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6efc004260 con 0x560f29c8b800 2024-04-23T18:19:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.372+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f0800fc70 con 0x560f29c8b800 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6efc004460 con 0x560f29c8b800 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 10 client.15867 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f29c9a380 con 0x560f29c8b800 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0800fc70 con 0x560f29c8b800 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6efc03d570 con 0x560f29c8b800 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 10 client.15867 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 10 client.15867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 20 client.15867 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29c9b840 0x560f29c9dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 4 client.15867 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f29bf3390 con 0x560f29c9b840 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.373+0000 7f6f10b23640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29c9b840 0x560f29c9dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.374+0000 7f6f10b23640 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29c9b840 0x560f29c9dc30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d6da9d77f9db4d0 server_cookie=ba69166828dac01c in_seq=0 out_seq=0 2024-04-23T18:19:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.374+0000 7f6eebfff640 10 client.15867 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6eebfff640 1 -- 172.21.15.26:0/3546090755 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f29bf3390 con 0x560f29c9b840 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6eebfff640 10 client.15867 handle_command_reply: tid=0 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 client.15867 shutdown 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 msgr2=0x560f29c889f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6efc005c40 tx=0x7f6efc02a480 comp rx=0 tx=0).stop 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 shutdown_connections 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29c9b840 0x560f29c9dc30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c8b800 0x560f29c889f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c80f40 0x560f29c88470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6eebfff640 0 client.15867 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 --2- 172.21.15.26:0/3546090755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c8b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 >> 172.21.15.26:0/3546090755 conn(0x560f29c7cb30 msgr2=0x560f29c82860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 shutdown_connections 2024-04-23T18:19:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 1 -- 172.21.15.26:0/3546090755 wait complete. 2024-04-23T18:19:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.375+0000 7f6f15b79740 20 client.15867 trim_cache size 0 max 16384 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 Processor -- start 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 -- start start 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c7bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29ca3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1ae220 con 0x560f29c80620 2024-04-23T18:19:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1aba30 con 0x560f29c80f40 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f15b79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1a9880 con 0x560f29c8b800 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50070/0 (socket says 172.21.15.26:50070) 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 -- 172.21.15.26:0/1626832855 learned_addr learned my addr 172.21.15.26:0/1626832855 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f10b23640 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c7bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 -- 172.21.15.26:0/1626832855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 msgr2=0x560f29ca3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29ca3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 -- 172.21.15.26:0/1626832855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 msgr2=0x560f29c7bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c7bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f03fff640 1 -- 172.21.15.26:0/1626832855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f29c80d70 con 0x560f29c80620 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.390+0000 7f6f10b23640 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c7bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f03fff640 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6efc03d9c0 tx=0x7f6efc031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847b0cc94320ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6ef27fc640 1 -- 172.21.15.26:0/1626832855 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc059030 con 0x560f29c80620 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6ef27fc640 1 -- 172.21.15.26:0/1626832855 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6efc004260 con 0x560f29c80620 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6ef27fc640 1 -- 172.21.15.26:0/1626832855 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc035040 con 0x560f29c80620 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 -- 172.21.15.26:0/1626832855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 msgr2=0x560f29c972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6efc03d9c0 tx=0x7f6efc031400 comp rx=0 tx=0).stop 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 -- 172.21.15.26:0/1626832855 shutdown_connections 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80f40 0x560f29c7bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29ca3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 --2- 172.21.15.26:0/1626832855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80620 0x560f29c972f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.391+0000 7f6f15b79740 1 -- 172.21.15.26:0/1626832855 >> 172.21.15.26:0/1626832855 conn(0x560f29c7cb30 msgr2=0x560f29c7b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- 172.21.15.26:0/1626832855 shutdown_connections 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- 172.21.15.26:0/1626832855 wait complete. 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- start start 2024-04-23T18:19:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 Processor -- start 2024-04-23T18:19:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c93c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29c915b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1a9ef0 con 0x560f29c80f40 2024-04-23T18:19:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1ab360 con 0x560f29c80620 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1aa5c0 con 0x560f29c8b800 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f15b79740 10 client.0 _ll_register_callbacks cb 0x7f6efc031580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f10b23640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c93c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f10b23640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50082/0 (socket says 172.21.15.26:50082) 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c93c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35280/0 (socket says 172.21.15.26:35280) 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.392+0000 7f6f10b23640 1 -- 172.21.15.26:0/2155402607 learned_addr learned my addr 172.21.15.26:0/2155402607 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f03fff640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29c915b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 -- 172.21.15.26:0/2155402607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 msgr2=0x560f29c915b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29c915b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 -- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 msgr2=0x560f29c93c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c93c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 -- 172.21.15.26:0/2155402607 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f29c91de0 con 0x560f29c80f40 2024-04-23T18:19:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f10b23640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6f08021040 tx=0x7f6f08009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65d0b87d3cfbc28 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6eebfff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08023040 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6f08005390 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6efc004260 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f08039050 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f15b79740 10 client.15873 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f29d01ea0 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.393+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6efc004260 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6f0802b0f0 con 0x560f29c80f40 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 10 client.15873 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 20 client.15873 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29d023f0 0x560f29d04810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 4 client.15873 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f29bed200 con 0x560f29d023f0 2024-04-23T18:19:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f03fff640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29d023f0 0x560f29d04810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6f03fff640 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29d023f0 0x560f29d04810 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6fce98bb0a0df0 server_cookie=e28eb6bb04c63d95 in_seq=0 out_seq=0 2024-04-23T18:19:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.394+0000 7f6eebfff640 10 client.15873 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6eebfff640 1 -- 172.21.15.26:0/2155402607 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f29bed200 con 0x560f29d023f0 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6eebfff640 10 client.15873 handle_command_reply: tid=0 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 client.15873 shutdown 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 msgr2=0x560f29c91010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6f08021040 tx=0x7f6f08009310 comp rx=0 tx=0).stop 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 shutdown_connections 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560f29d023f0 0x560f29d04810 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f29c8b800 0x560f29c915b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560f29c80f40 0x560f29c91010 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6eebfff640 0 client.15873 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:11.436 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 --2- 172.21.15.26:0/2155402607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560f29c80620 0x560f29c93c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 >> 172.21.15.26:0/2155402607 conn(0x560f29c7cb30 msgr2=0x560f29c7b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 shutdown_connections 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 1 -- 172.21.15.26:0/2155402607 wait complete. 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.442+0000 7f6f15b79740 20 client.15873 trim_cache size 0 max 16384 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:19:11.437 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 msgr2=0x7f6ef40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 0x7f6ef40a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f0409c9d0 tx=0x7f6f0406b680 comp rx=0 tx=0).stop 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 msgr2=0x7f6f0c19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400eb10 tx=0x7f6f0400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 shutdown_connections 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f6ef40a3dc0 0x7f6ef40a6280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6f0c1080e0 0x7f6f0c19cec0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f0c06b120 0x7f6f0c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 --2- 172.21.15.26:0/84652744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f0c06a780 0x7f6f0c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 >> 172.21.15.26:0/84652744 conn(0x7f6f0c076c50 msgr2=0x7f6f0c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 shutdown_connections 2024-04-23T18:19:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:11.443+0000 7f6f12d87640 1 -- 172.21.15.26:0/84652744 wait complete. 2024-04-23T18:19:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:10 smithi110 ceph-mon[32585]: pgmap v64: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:10 smithi026 ceph-mon[29364]: pgmap v64: 97 pgs: 97 active+clean; 59 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 211 op/s 2024-04-23T18:19:11.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 103196 8.72 MB/sec warmup 66 sec latency 13.667 ms 2024-04-23T18:19:12.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 104867 8.67 MB/sec warmup 67 sec latency 43.551 ms 2024-04-23T18:19:13.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:12 smithi131 ceph-mon[33041]: pgmap v65: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.9 MiB/s wr, 325 op/s 2024-04-23T18:19:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:12 smithi110 ceph-mon[32585]: pgmap v65: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.9 MiB/s wr, 325 op/s 2024-04-23T18:19:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:12 smithi026 ceph-mon[29364]: pgmap v65: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.9 MiB/s wr, 325 op/s 2024-04-23T18:19:13.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 106897 8.75 MB/sec warmup 68 sec latency 28.626 ms 2024-04-23T18:19:14.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 108892 8.72 MB/sec warmup 69 sec latency 85.137 ms 2024-04-23T18:19:15.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:14 smithi131 ceph-mon[33041]: pgmap v66: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-04-23T18:19:15.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:14 smithi110 ceph-mon[32585]: pgmap v66: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-04-23T18:19:15.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:14 smithi026 ceph-mon[29364]: pgmap v66: 97 pgs: 97 active+clean; 65 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-04-23T18:19:15.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 110602 8.77 MB/sec warmup 70 sec latency 16.991 ms 2024-04-23T18:19:15.876 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-04-23T18:19:15.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.995+0000 7effeeb19640 1 -- 172.21.15.26:0/1899926163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe8106170 msgr2=0x7effe810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.995+0000 7effeeb19640 1 --2- 172.21.15.26:0/1899926163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe8106170 0x7effe810dd40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7effe0009ca0 tx=0x7effe002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 -- 172.21.15.26:0/1899926163 shutdown_connections 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 --2- 172.21.15.26:0/1899926163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe8101a50 0x7effe8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 --2- 172.21.15.26:0/1899926163 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe8106170 0x7effe810dd40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 --2- 172.21.15.26:0/1899926163 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe81010a0 0x7effe8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 -- 172.21.15.26:0/1899926163 >> 172.21.15.26:0/1899926163 conn(0x7effe8077f50 msgr2=0x7effe8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 -- 172.21.15.26:0/1899926163 shutdown_connections 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 -- 172.21.15.26:0/1899926163 wait complete. 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 Processor -- start 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.996+0000 7effeeb19640 1 -- start start 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe8106170 0x7effe8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe819cfe0 0x7effe819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7effe8111110 con 0x7effe819cfe0 2024-04-23T18:19:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7effe8110f90 con 0x7effe81010a0 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effeeb19640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7effe8111290 con 0x7effe8106170 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effed0b6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe819cfe0 0x7effe819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35300/0 (socket says 172.21.15.26:35300) 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 -- 172.21.15.26:0/3047799116 learned_addr learned my addr 172.21.15.26:0/3047799116 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effe7fff640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe8106170 0x7effe8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 -- 172.21.15.26:0/3047799116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe8106170 msgr2=0x7effe8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe8106170 0x7effe8198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 -- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe819cfe0 msgr2=0x7effe819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe819cfe0 0x7effe819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.997+0000 7effec8b5640 1 -- 172.21.15.26:0/3047799116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe819dad0 con 0x7effe81010a0 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effec8b5640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effdc00e9f0 tx=0x7effdc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc00cd80 con 0x7effe81010a0 2024-04-23T18:19:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effdc0108e0 con 0x7effe81010a0 2024-04-23T18:19:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe819ddc0 con 0x7effe81010a0 2024-04-23T18:19:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc018980 con 0x7effe81010a0 2024-04-23T18:19:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.998+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe81a55b0 con 0x7effe81010a0 2024-04-23T18:19:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:15.999+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe8068d70 con 0x7effe81010a0 2024-04-23T18:19:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.000+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7effdc010430 con 0x7effe81010a0 2024-04-23T18:19:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.001+0000 7effe5ffb640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 0x7effd80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.001+0000 7effe7fff640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 0x7effd80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.001+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7effdc0d0ef0 con 0x7effe81010a0 2024-04-23T18:19:16.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.002+0000 7effe7fff640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 0x7effd80a6330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effe8100780 tx=0x7effd4005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.005+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effdc09cf90 con 0x7effe81010a0 2024-04-23T18:19:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:15 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:16.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.239+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effe8199a70 con 0x7effe81010a0 2024-04-23T18:19:16.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.240+0000 7effe5ffb640 1 -- 172.21.15.26:0/3047799116 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11890 (secure 0 0 0) 0x7effe8199a70 con 0x7effe81010a0 2024-04-23T18:19:16.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:19:16.233 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:18:29.409119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:19:16.234 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 28 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 msgr2=0x7effd80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 0x7effd80a6330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effe8100780 tx=0x7effd4005f90 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 msgr2=0x7effe8198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effdc00e9f0 tx=0x7effdc00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 shutdown_connections 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7effd80a3e70 0x7effd80a6330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7effe819cfe0 0x7effe819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effe8106170 0x7effe8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 --2- 172.21.15.26:0/3047799116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effe81010a0 0x7effe8198720 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 >> 172.21.15.26:0/3047799116 conn(0x7effe8077f50 msgr2=0x7effe80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 shutdown_connections 2024-04-23T18:19:16.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.243+0000 7effeeb19640 1 -- 172.21.15.26:0/3047799116 wait complete. 2024-04-23T18:19:16.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:15 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:16.251 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 86.5 --await-for 42.0' 2024-04-23T18:19:16.251 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 86.5 --await-for 42.0 2024-04-23T18:19:16.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 -- 172.21.15.26:0/1299108969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f8101ac0 msgr2=0x7f61f8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 --2- 172.21.15.26:0/1299108969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f8101ac0 0x7f61f8105ab0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009900 tx=0x7f61ec02a410 comp rx=0 tx=0).stop 2024-04-23T18:19:16.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 -- 172.21.15.26:0/1299108969 shutdown_connections 2024-04-23T18:19:16.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 --2- 172.21.15.26:0/1299108969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 --2- 172.21.15.26:0/1299108969 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f8101ac0 0x7f61f8105ab0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 --2- 172.21.15.26:0/1299108969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 -- 172.21.15.26:0/1299108969 >> 172.21.15.26:0/1299108969 conn(0x7f61f8077fb0 msgr2=0x7f61f80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 -- 172.21.15.26:0/1299108969 shutdown_connections 2024-04-23T18:19:16.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.381+0000 7f61fe005640 1 -- 172.21.15.26:0/1299108969 wait complete. 2024-04-23T18:19:16.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 Processor -- start 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 -- start start 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f819d010 0x7f61f819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81110b0 con 0x7f61f819d010 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8110f30 con 0x7f61f8106190 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61fe005640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8111230 con 0x7f61f8101110 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35326/0 (socket says 172.21.15.26:35326) 2024-04-23T18:19:16.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:57408/0 (socket says 172.21.15.26:57408) 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f6ffd640 1 -- 172.21.15.26:0/3686362372 learned_addr learned my addr 172.21.15.26:0/3686362372 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.382+0000 7f61f7fff640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f819d010 0x7f61f819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 -- 172.21.15.26:0/3686362372 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 msgr2=0x7f61f81987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 -- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f819d010 msgr2=0x7f61f819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f819d010 0x7f61f819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 -- 172.21.15.26:0/3686362372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f819db00 con 0x7f61f8106190 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f77fe640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f61ec032fd0 tx=0x7f61ec031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f4ff9640 1 -- 172.21.15.26:0/3686362372 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec03f030 con 0x7f61f8106190 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f4ff9640 1 -- 172.21.15.26:0/3686362372 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61ec004490 con 0x7f61f8106190 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f819dd90 con 0x7f61f8106190 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61f4ff9640 1 -- 172.21.15.26:0/3686362372 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec035070 con 0x7f61f8106190 2024-04-23T18:19:16.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.383+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f81a5550 con 0x7f61f8106190 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.386+0000 7f6200df9740 1 Processor -- start 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.386+0000 7f61f4ff9640 1 -- 172.21.15.26:0/3686362372 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f61ec042030 con 0x7f61f8106190 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.386+0000 7f6200df9740 1 -- start start 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e838360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 0x56177e849890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56177e79d370 con 0x56177e8388a0 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56177e73a2e0 con 0x56177e837f80 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f6200df9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56177e799330 con 0x56177e8474a0 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e838360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50122/0 (socket says 172.21.15.26:50122) 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f77fe640 1 -- 172.21.15.26:0/641458961 learned_addr learned my addr 172.21.15.26:0/641458961 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 0x56177e849890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 -- 172.21.15.26:0/641458961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 msgr2=0x56177e849890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 0x56177e849890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 -- 172.21.15.26:0/641458961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 msgr2=0x56177e846f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f7fff640 1 -- 172.21.15.26:0/641458961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177e57bd80 con 0x56177e837f80 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.387+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 0x56177e849890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61f4ff9640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 0x7f61e00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61f7fff640 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e838360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009ef0 tx=0x7f61f0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de32fa08d0b25603 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61f77fe640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 0x7f61e00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61f4ff9640 1 -- 172.21.15.26:0/3686362372 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f61ec0f0b70 con 0x7f61f8106190 2024-04-23T18:19:16.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61ddffb640 1 -- 172.21.15.26:0/641458961 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0004b20 con 0x56177e837f80 2024-04-23T18:19:16.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61ddffb640 1 -- 172.21.15.26:0/641458961 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61f0004cc0 con 0x56177e837f80 2024-04-23T18:19:16.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.388+0000 7f61ddffb640 1 -- 172.21.15.26:0/641458961 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f000e6b0 con 0x56177e837f80 2024-04-23T18:19:16.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 -- 172.21.15.26:0/641458961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 msgr2=0x56177e838360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f61f77fe640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 0x7f61e00a6260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61e80058c0 tx=0x7f61e8005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e838360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009ef0 tx=0x7f61f0011810 comp rx=0 tx=0).stop 2024-04-23T18:19:16.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 -- 172.21.15.26:0/641458961 shutdown_connections 2024-04-23T18:19:16.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8474a0 0x56177e849890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 --2- 172.21.15.26:0/641458961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e838360 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 -- 172.21.15.26:0/641458961 >> 172.21.15.26:0/641458961 conn(0x56177e8343c0 msgr2=0x56177e8347a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 -- 172.21.15.26:0/641458961 shutdown_connections 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.389+0000 7f6200df9740 1 -- 172.21.15.26:0/641458961 wait complete. 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 -- start start 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 Processor -- start 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 0x56177e844970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56177e84a2e0 con 0x56177e8474a0 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56177e737fb0 con 0x56177e837f80 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56177e84a460 con 0x56177e8388a0 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f6200df9740 10 client.0 _ll_register_callbacks cb 0x56177e7379a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50136/0 (socket says 172.21.15.26:50136) 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f7fff640 1 -- 172.21.15.26:0/837702479 learned_addr learned my addr 172.21.15.26:0/837702479 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35336/0 (socket says 172.21.15.26:35336) 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.390+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 0x56177e844970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 -- 172.21.15.26:0/837702479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 msgr2=0x56177e844970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 0x56177e844970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 -- 172.21.15.26:0/837702479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 msgr2=0x56177e844eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 -- 172.21.15.26:0/837702479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177e5dc510 con 0x56177e837f80 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f7fff640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 0x56177e844970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61f77fe640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61e8012fa0 tx=0x7f61e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1516425c7c8d600 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8004260 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f61e8004400 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61f0004cc0 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61e80118c0 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f6200df9740 10 client.25391 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.391+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56177e46f670 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e80035f0 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f61e8027020 con 0x56177e837f80 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 10 client.25391 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 20 client.25391 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e859cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 4 client.25391 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f61f0004cc0 con 0x56177e857900 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.392+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e859cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e859cf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01577ae2276083a server_cookie=2c8de235fbc5b69c in_seq=0 out_seq=0 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f61d6ffd640 10 client.25391 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f61d6ffd640 1 -- 172.21.15.26:0/837702479 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f61f0004cc0 con 0x56177e857900 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f61d6ffd640 10 client.25391 handle_command_reply: tid=0 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f6200df9740 1 client.25391 shutdown 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 msgr2=0x56177e844430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61e8012fa0 tx=0x7f61e8004730 comp rx=0 tx=0).stop 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 shutdown_connections 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.393+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e859cf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f61d6ffd640 0 client.25391 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e844eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e8388a0 0x56177e844970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 --2- 172.21.15.26:0/837702479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e837f80 0x56177e844430 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 >> 172.21.15.26:0/837702479 conn(0x56177e8343c0 msgr2=0x56177e83a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 shutdown_connections 2024-04-23T18:19:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 1 -- 172.21.15.26:0/837702479 wait complete. 2024-04-23T18:19:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.394+0000 7f6200df9740 20 client.25391 trim_cache size 0 max 16384 2024-04-23T18:19:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 Processor -- start 2024-04-23T18:19:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 -- start start 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e853270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8474a0 0x56177e85f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81ae2a0 con 0x56177e8388a0 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81abab0 con 0x56177e8474a0 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f6200df9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81a9940 con 0x56177e837f80 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50146/0 (socket says 172.21.15.26:50146) 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f6ffd640 1 -- 172.21.15.26:0/1558007701 learned_addr learned my addr 172.21.15.26:0/1558007701 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e853270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f7fff640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8474a0 0x56177e85f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 -- 172.21.15.26:0/1558007701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8474a0 msgr2=0x56177e85f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8474a0 0x56177e85f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 -- 172.21.15.26:0/1558007701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 msgr2=0x56177e846370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f77fe640 1 -- 172.21.15.26:0/1558007701 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177e83f270 con 0x56177e837f80 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.412+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.413+0000 7f61f77fe640 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e853270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61e8019040 tx=0x7f61e8032620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96ea274a71cce161 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.413+0000 7f61dd7fa640 1 -- 172.21.15.26:0/1558007701 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8004460 con 0x56177e837f80 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.413+0000 7f61dd7fa640 1 -- 172.21.15.26:0/1558007701 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e8037070 con 0x56177e837f80 2024-04-23T18:19:16.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.413+0000 7f61dd7fa640 1 -- 172.21.15.26:0/1558007701 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e80327f0 con 0x56177e837f80 2024-04-23T18:19:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- 172.21.15.26:0/1558007701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 msgr2=0x56177e853270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e853270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61e8019040 tx=0x7f61e8032620 comp rx=0 tx=0).stop 2024-04-23T18:19:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- 172.21.15.26:0/1558007701 shutdown_connections 2024-04-23T18:19:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8474a0 0x56177e85f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8388a0 0x56177e846370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 --2- 172.21.15.26:0/1558007701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e853270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- 172.21.15.26:0/1558007701 >> 172.21.15.26:0/1558007701 conn(0x56177e8343c0 msgr2=0x56177e845c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- 172.21.15.26:0/1558007701 shutdown_connections 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- 172.21.15.26:0/1558007701 wait complete. 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 -- start start 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 Processor -- start 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.414+0000 7f6200df9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8388a0 0x56177e84b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e84bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81a9f90 con 0x56177e8474a0 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81ab3e0 con 0x56177e8388a0 2024-04-23T18:19:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81aa640 con 0x56177e837f80 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e84bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 10 client.0 _ll_register_callbacks cb 0x7f61e800f840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:57434/0 (socket says 172.21.15.26:57434) 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 -- 172.21.15.26:0/730992204 learned_addr learned my addr 172.21.15.26:0/730992204 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8388a0 0x56177e84b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 -- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8388a0 msgr2=0x56177e84b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8388a0 0x56177e84b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 -- 172.21.15.26:0/730992204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 msgr2=0x56177e84bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e84bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 -- 172.21.15.26:0/730992204 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177e8bd0d0 con 0x56177e837f80 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61f77fe640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009ce0 tx=0x7f61e8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61466e72a8e2309f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:19:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8011c60 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f61e8005020 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.415+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61e8037070 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61e8004530 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 10 client.25381 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56177e8be3b0 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e8037070 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f61e8027040 con 0x56177e837f80 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 10 client.25381 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 10 client.25381 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 20 client.25381 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e8c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 4 client.25381 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.5, "await_for": 42.0}] 2024-04-23T18:19:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.5, "await_for": 42.0}) -- 0x56177e7aad90 con 0x56177e857900 2024-04-23T18:19:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.416+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e8c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.417+0000 7f61f6ffd640 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e8c0a10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68004dd937ee3ea6 server_cookie=cf3d0d45a573cba7 in_seq=0 out_seq=0 2024-04-23T18:19:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.417+0000 7f61d6ffd640 10 client.25381 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:16.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:15 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:16.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f61d6ffd640 1 -- 172.21.15.26:0/730992204 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56177e7aad90 con 0x56177e857900 2024-04-23T18:19:16.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f61d6ffd640 10 client.25381 handle_command_reply: tid=0 2024-04-23T18:19:16.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 client.25381 shutdown 2024-04-23T18:19:16.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 msgr2=0x56177e84b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009ce0 tx=0x7f61e8017000 comp rx=0 tx=0).stop 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 shutdown_connections 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56177e857900 0x56177e8c0a10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56177e8388a0 0x56177e84b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f61d6ffd640 0 client.25381 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56177e8474a0 0x56177e84bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 --2- 172.21.15.26:0/730992204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56177e837f80 0x56177e84b490 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 >> 172.21.15.26:0/730992204 conn(0x56177e8343c0 msgr2=0x56177e8464f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.492+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 shutdown_connections 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f6200df9740 1 -- 172.21.15.26:0/730992204 wait complete. 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f6200df9740 20 client.25381 trim_cache size 0 max 16384 2024-04-23T18:19:16.487 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 28, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 14, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "780d2ace": { 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 14, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 86.5, 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:19:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:16.489 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 msgr2=0x7f61e00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 0x7f61e00a6260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61e80058c0 tx=0x7f61e8005f90 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 msgr2=0x7f61f8198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f61ec032fd0 tx=0x7f61ec031a20 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 shutdown_connections 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f61e00a3da0 0x7f61e00a6260 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f8106190 0x7f61f8198ce0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f61f819d010 0x7f61f819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 --2- 172.21.15.26:0/3686362372 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61f8101110 0x7f61f81987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 >> 172.21.15.26:0/3686362372 conn(0x7f61f8077fb0 msgr2=0x7f61f806a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 shutdown_connections 2024-04-23T18:19:16.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:16.493+0000 7f61fe005640 1 -- 172.21.15.26:0/3686362372 wait complete. 2024-04-23T18:19:16.497 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '780d2ace', quiesce took 0.6 seconds. Will release after: 26.5 2024-04-23T18:19:16.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.75 MB/sec warmup 71 sec latency 235.990 ms 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: pgmap v67: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 386 op/s 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3047799116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:16 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: pgmap v67: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 386 op/s 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3047799116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:16 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: pgmap v67: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 386 op/s 2024-04-23T18:19:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3047799116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:16 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:17.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.62 MB/sec warmup 72 sec latency 1236.135 ms 2024-04-23T18:19:18.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:17 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:18.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:17 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:18.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:17 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:17 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:17 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:17 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:17 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:17 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:17 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:19:18.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.51 MB/sec warmup 73 sec latency 2236.253 ms 2024-04-23T18:19:19.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:19 smithi110 ceph-mon[32585]: pgmap v68: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:19 smithi026 ceph-mon[29364]: pgmap v68: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:19.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:19 smithi131 ceph-mon[33041]: pgmap v68: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:19.660 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.39 MB/sec warmup 74 sec latency 3236.403 ms 2024-04-23T18:19:20.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.28 MB/sec warmup 75 sec latency 4236.491 ms 2024-04-23T18:19:21.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:21 smithi110 ceph-mon[32585]: pgmap v69: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:21 smithi026 ceph-mon[29364]: pgmap v69: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:21 smithi131 ceph-mon[33041]: pgmap v69: 97 pgs: 97 active+clean; 65 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.2 MiB/s wr, 277 op/s 2024-04-23T18:19:21.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.17 MB/sec warmup 76 sec latency 5236.571 ms 2024-04-23T18:19:22.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 8.06 MB/sec warmup 77 sec latency 6236.705 ms 2024-04-23T18:19:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:23 smithi110 ceph-mon[32585]: pgmap v70: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 319 op/s 2024-04-23T18:19:23.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:23 smithi026 ceph-mon[29364]: pgmap v70: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 319 op/s 2024-04-23T18:19:23.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:23 smithi131 ceph-mon[33041]: pgmap v70: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 319 op/s 2024-04-23T18:19:23.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.96 MB/sec warmup 78 sec latency 7236.822 ms 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 -- 172.21.15.26:0/20793641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc101130 msgr2=0x7fdddc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 --2- 172.21.15.26:0/20793641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc101130 0x7fdddc1015b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009930 tx=0x7fddd002a420 comp rx=0 tx=0).stop 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 -- 172.21.15.26:0/20793641 shutdown_connections 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 --2- 172.21.15.26:0/20793641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 --2- 172.21.15.26:0/20793641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc101130 0x7fdddc1015b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 --2- 172.21.15.26:0/20793641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc106f90 0x7fdddc107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 -- 172.21.15.26:0/20793641 >> 172.21.15.26:0/20793641 conn(0x7fdddc077c30 msgr2=0x7fdddc078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 -- 172.21.15.26:0/20793641 shutdown_connections 2024-04-23T18:19:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.543+0000 7fdde17fe640 1 -- 172.21.15.26:0/20793641 wait complete. 2024-04-23T18:19:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 Processor -- start 2024-04-23T18:19:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 -- start start 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc101130 0x7fdddc102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc1030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1110a0 con 0x7fdddc106f90 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc110f20 con 0x7fdddc101130 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdde17fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc111220 con 0x7fdddc101af0 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdddb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdddb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47630/0 (socket says 172.21.15.26:47630) 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdddb7fe640 1 -- 172.21.15.26:0/2210590131 learned_addr learned my addr 172.21.15.26:0/2210590131 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fdddaffd640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc101130 0x7fdddc102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.544+0000 7fddda7fc640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc1030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 -- 172.21.15.26:0/2210590131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 msgr2=0x7fdddc1030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc1030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 -- 172.21.15.26:0/2210590131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc101130 msgr2=0x7fdddc102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc101130 0x7fdddc102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 -- 172.21.15.26:0/2210590131 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc1a6b90 con 0x7fdddc106f90 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fddda7fc640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdddb7fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e970 tx=0x7fddd400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fddc3fff640 1 -- 172.21.15.26:0/2210590131 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd400ccc0 con 0x7fdddc106f90 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdddc1a6e80 con 0x7fdddc106f90 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fddc3fff640 1 -- 172.21.15.26:0/2210590131 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd400ce60 con 0x7fdddc106f90 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fddc3fff640 1 -- 172.21.15.26:0/2210590131 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4010430 con 0x7fdddc106f90 2024-04-23T18:19:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.545+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdddc1a7350 con 0x7fdddc106f90 2024-04-23T18:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 Processor -- start 2024-04-23T18:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fddc3fff640 1 -- 172.21.15.26:0/2210590131 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fddd4004f90 con 0x7fdddc106f90 2024-04-23T18:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 -- start start 2024-04-23T18:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5620 0x5586e75e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75f0800 0x5586e75f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5586e754a970 con 0x5586e75e5f40 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5586e74e74a0 con 0x5586e75e5620 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdde45ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5586e7546930 con 0x5586e75f0800 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdddaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdddaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47642/0 (socket says 172.21.15.26:47642) 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fdddaffd640 1 -- 172.21.15.26:0/2874403168 learned_addr learned my addr 172.21.15.26:0/2874403168 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fddda7fc640 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75f0800 0x5586e75f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fddc3fff640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 0x7fddc40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.548+0000 7fddda7fc640 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5620 0x5586e75e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 -- 172.21.15.26:0/2874403168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75f0800 msgr2=0x5586e75f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75f0800 0x5586e75f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 -- 172.21.15.26:0/2874403168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5620 msgr2=0x5586e75e5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5620 0x5586e75e5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddb7fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 0x7fddc40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 -- 172.21.15.26:0/2874403168 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586e725d210 con 0x5586e75e5f40 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fddc3fff640 1 -- 172.21.15.26:0/2210590131 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fddd40d0c50 con 0x7fdddc106f90 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddaffd640 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fddcc010370 tx=0x7fddcc010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6aa7d5beb81b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fddc17fa640 1 -- 172.21.15.26:0/2874403168 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc016070 con 0x5586e75e5f40 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fdddb7fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 0x7fddc40a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fddd40ce480 tx=0x7fddd40920b0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fddc17fa640 1 -- 172.21.15.26:0/2874403168 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc004d20 con 0x5586e75e5f40 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.549+0000 7fddc17fa640 1 -- 172.21.15.26:0/2874403168 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc005360 con 0x5586e75e5f40 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- 172.21.15.26:0/2874403168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 msgr2=0x5586e75f02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fddcc010370 tx=0x7fddcc010830 comp rx=0 tx=0).stop 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- 172.21.15.26:0/2874403168 shutdown_connections 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75f0800 0x5586e75f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e75f02c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- 172.21.15.26:0/2874403168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5620 0x5586e75e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- 172.21.15.26:0/2874403168 >> 172.21.15.26:0/2874403168 conn(0x5586e75e1b30 msgr2=0x5586e75e1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- 172.21.15.26:0/2874403168 shutdown_connections 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- 172.21.15.26:0/2874403168 wait complete. 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- start start 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 Processor -- start 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 0x5586e75ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e75ed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5586e75f3430 con 0x5586e75e5620 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5586e74e50b0 con 0x5586e75f0800 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.550+0000 7fdde45ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5586e75f35b0 con 0x5586e75e5f40 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fdde45ee740 10 client.0 _ll_register_callbacks cb 0x5586e74d2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47644/0 (socket says 172.21.15.26:47644) 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 -- 172.21.15.26:0/781369619 learned_addr learned my addr 172.21.15.26:0/781369619 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fdddaffd640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 0x5586e75ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 -- 172.21.15.26:0/781369619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 msgr2=0x5586e75ed3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 0x5586e75ed3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 -- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 msgr2=0x5586e75ed8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e75ed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 -- 172.21.15.26:0/781369619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586e7329620 con 0x5586e75e5620 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fdddaffd640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 0x5586e75ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddda7fc640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fddd0005ec0 tx=0x7fddd0032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427287fe50b3942d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0035040 con 0x5586e75e5620 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fddd0004240 con 0x5586e75e5620 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.551+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddcc004d20 con 0x5586e75e5620 2024-04-23T18:19:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddd0004440 con 0x5586e75e5620 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 10 client.15909 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586e75ff2a0 con 0x5586e75e5620 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc004d20 con 0x5586e75e5620 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fddd003a040 con 0x5586e75e5620 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 20 client.15909 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7600760 0x5586e7602b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 4 client.15909 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586e7558390 con 0x5586e7600760 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.552+0000 7fdddaffd640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7600760 0x5586e7602b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.553+0000 7fddbeff5640 10 client.15909 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.553+0000 7fdddaffd640 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7600760 0x5586e7602b90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=677a13eae17c73ab server_cookie=7c86fd3e129df78c in_seq=0 out_seq=0 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.553+0000 7fddbeff5640 1 -- 172.21.15.26:0/781369619 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586e7558390 con 0x5586e7600760 2024-04-23T18:19:24.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.553+0000 7fddbeff5640 10 client.15909 handle_command_reply: tid=0 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.553+0000 7fdde45ee740 1 client.15909 shutdown 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 msgr2=0x5586e75f00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fddd0005ec0 tx=0x7fddd0032f00 comp rx=0 tx=0).stop 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 shutdown_connections 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7600760 0x5586e7602b90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e75ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5f40 0x5586e75ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 --2- 172.21.15.26:0/781369619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5620 0x5586e75f00b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 >> 172.21.15.26:0/781369619 conn(0x5586e75e1b30 msgr2=0x5586e75e78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fddbeff5640 0 client.15909 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 shutdown_connections 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 1 -- 172.21.15.26:0/781369619 wait complete. 2024-04-23T18:19:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.554+0000 7fdde45ee740 20 client.15909 trim_cache size 0 max 16384 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 Processor -- start 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 -- start start 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e76089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1b2700 con 0x5586e75e5f40 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1aff10 con 0x5586e75f0800 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdde45ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1add60 con 0x5586e75e5620 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47656/0 (socket says 172.21.15.26:47656) 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 -- 172.21.15.26:0/3019892045 learned_addr learned my addr 172.21.15.26:0/3019892045 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:24.567 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 -- 172.21.15.26:0/3019892045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 msgr2=0x5586e75fc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e76089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 -- 172.21.15.26:0/3019892045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 msgr2=0x5586e76089e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e76089e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.569+0000 7fdddaffd640 1 -- 172.21.15.26:0/3019892045 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586e75eb8f0 con 0x5586e75e5f40 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e76089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5586e75e6bc0 tx=0x7fddcc009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68da21a5ee471e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fddc0ff9640 1 -- 172.21.15.26:0/3019892045 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc016040 con 0x5586e75e5f40 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fddc0ff9640 1 -- 172.21.15.26:0/3019892045 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc0297d0 con 0x5586e75e5f40 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fddc0ff9640 1 -- 172.21.15.26:0/3019892045 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc012d40 con 0x5586e75e5f40 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 -- 172.21.15.26:0/3019892045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 msgr2=0x5586e76084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5586e75e6bc0 tx=0x7fddcc009680 comp rx=0 tx=0).stop 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 -- 172.21.15.26:0/3019892045 shutdown_connections 2024-04-23T18:19:24.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75f0800 0x5586e76089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75e5f40 0x5586e76084a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3019892045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 -- 172.21.15.26:0/3019892045 >> 172.21.15.26:0/3019892045 conn(0x5586e75e1b30 msgr2=0x5586e75e0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.570+0000 7fdde45ee740 1 -- 172.21.15.26:0/3019892045 shutdown_connections 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 -- 172.21.15.26:0/3019892045 wait complete. 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 -- start start 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 Processor -- start 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 0x5586e75f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1ae3d0 con 0x5586e75f0800 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1af840 con 0x5586e75e5f40 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1aeaa0 con 0x5586e75e5620 2024-04-23T18:19:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdde45ee740 10 client.0 _ll_register_callbacks cb 0x7fddd0032dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fddda7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fddda7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47672/0 (socket says 172.21.15.26:47672) 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fddda7fc640 1 -- 172.21.15.26:0/3660234208 learned_addr learned my addr 172.21.15.26:0/3660234208 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 0x5586e75f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.571+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 -- 172.21.15.26:0/3660234208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 msgr2=0x5586e75f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 -- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 msgr2=0x5586e75f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 0x5586e75f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 -- 172.21.15.26:0/3660234208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586e7665300 con 0x5586e75f0800 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 0x5586e75f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddda7fc640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fddd002a940 tx=0x7fddd0032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986c9773a0e6cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0041070 con 0x5586e75f0800 2024-04-23T18:19:24.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fddd0049030 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddcc0297d0 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddd0064050 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 10 client.15915 fetch_fsmap learned FSMap version 28 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586e7666560 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc0297d0 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fddd0048980 con 0x5586e75f0800 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 10 client.15915 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 20 client.15915 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7666c90 0x5586e76690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 4 client.15915 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.572+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5586e7552200 con 0x5586e7666c90 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.573+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7666c90 0x5586e76690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.573+0000 7fdddaffd640 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7666c90 0x5586e76690b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccdd575afbb233ef server_cookie=8823472c67a0af8d in_seq=0 out_seq=0 2024-04-23T18:19:24.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.573+0000 7fddbeff5640 10 client.15915 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fddbeff5640 1 -- 172.21.15.26:0/3660234208 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5586e7552200 con 0x5586e7666c90 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fddbeff5640 10 client.15915 handle_command_reply: tid=0 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 client.15915 shutdown 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 msgr2=0x5586e7664d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fddd002a940 tx=0x7fddd0032e70 comp rx=0 tx=0).stop 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 shutdown_connections 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5586e7666c90 0x5586e76690b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5586e75e5f40 0x5586e75f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fddbeff5640 0 client.15915 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5586e75f0800 0x5586e7664d10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 --2- 172.21.15.26:0/3660234208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5586e75e5620 0x5586e75f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 >> 172.21.15.26:0/3660234208 conn(0x5586e75e1b30 msgr2=0x5586e75ed170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 shutdown_connections 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 1 -- 172.21.15.26:0/3660234208 wait complete. 2024-04-23T18:19:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.602+0000 7fdde45ee740 20 client.15915 trim_cache size 0 max 16384 2024-04-23T18:19:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:19:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:19:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:19:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 msgr2=0x7fddc40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 0x7fddc40a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fddd40ce480 tx=0x7fddd40920b0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 msgr2=0x7fdddc1035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e970 tx=0x7fddd400ed70 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 shutdown_connections 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddc40a3dc0 0x7fddc40a6280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdddc106f90 0x7fdddc1035e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdddc101af0 0x7fdddc1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 --2- 172.21.15.26:0/2210590131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdddc101130 0x7fdddc102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 >> 172.21.15.26:0/2210590131 conn(0x7fdddc077c30 msgr2=0x7fdddc06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 shutdown_connections 2024-04-23T18:19:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:24.603+0000 7fdde17fe640 1 -- 172.21.15.26:0/2210590131 wait complete. 2024-04-23T18:19:24.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.86 MB/sec warmup 79 sec latency 8236.915 ms 2024-04-23T18:19:25.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:25 smithi110 ceph-mon[32585]: pgmap v71: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 204 op/s 2024-04-23T18:19:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:25 smithi026 ceph-mon[29364]: pgmap v71: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 204 op/s 2024-04-23T18:19:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:25 smithi131 ceph-mon[33041]: pgmap v71: 97 pgs: 97 active+clean; 66 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 204 op/s 2024-04-23T18:19:25.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.76 MB/sec warmup 80 sec latency 9236.963 ms 2024-04-23T18:19:26.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.67 MB/sec warmup 81 sec latency 10237.070 ms 2024-04-23T18:19:27.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:27 smithi110 ceph-mon[32585]: pgmap v72: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-04-23T18:19:27.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:27 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:27 smithi026 ceph-mon[29364]: pgmap v72: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-04-23T18:19:27.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:27 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:27 smithi131 ceph-mon[33041]: pgmap v72: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-04-23T18:19:27.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:27 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:27.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.57 MB/sec warmup 82 sec latency 11237.194 ms 2024-04-23T18:19:28.661 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.48 MB/sec warmup 83 sec latency 12237.316 ms 2024-04-23T18:19:29.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:29 smithi110 ceph-mon[32585]: pgmap v73: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:29.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:29 smithi026 ceph-mon[29364]: pgmap v73: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:29.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:29 smithi131 ceph-mon[33041]: pgmap v73: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:29.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.39 MB/sec warmup 84 sec latency 13237.433 ms 2024-04-23T18:19:30.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.31 MB/sec warmup 85 sec latency 14237.537 ms 2024-04-23T18:19:31.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:31 smithi110 ceph-mon[32585]: pgmap v74: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:31 smithi026 ceph-mon[29364]: pgmap v74: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:31 smithi131 ceph-mon[33041]: pgmap v74: 97 pgs: 97 active+clean; 65 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 493 KiB/s wr, 65 op/s 2024-04-23T18:19:31.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.22 MB/sec warmup 86 sec latency 15237.577 ms 2024-04-23T18:19:32.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.14 MB/sec warmup 87 sec latency 16237.685 ms 2024-04-23T18:19:33.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:33 smithi110 ceph-mon[32585]: pgmap v75: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 669 KiB/s wr, 72 op/s 2024-04-23T18:19:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:33 smithi026 ceph-mon[29364]: pgmap v75: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 669 KiB/s wr, 72 op/s 2024-04-23T18:19:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:33 smithi131 ceph-mon[33041]: pgmap v75: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 669 KiB/s wr, 72 op/s 2024-04-23T18:19:33.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 7.06 MB/sec warmup 88 sec latency 17237.723 ms 2024-04-23T18:19:34.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.98 MB/sec warmup 89 sec latency 18237.822 ms 2024-04-23T18:19:35.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:35 smithi110 ceph-mon[32585]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 205 KiB/s wr, 31 op/s 2024-04-23T18:19:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:35 smithi026 ceph-mon[29364]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 205 KiB/s wr, 31 op/s 2024-04-23T18:19:35.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:35 smithi131 ceph-mon[33041]: pgmap v76: 97 pgs: 97 active+clean; 66 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 205 KiB/s wr, 31 op/s 2024-04-23T18:19:35.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.90 MB/sec warmup 90 sec latency 19237.859 ms 2024-04-23T18:19:36.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.82 MB/sec warmup 91 sec latency 20237.912 ms 2024-04-23T18:19:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:37 smithi110 ceph-mon[32585]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 922 KiB/s wr, 54 op/s 2024-04-23T18:19:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:37 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:37 smithi026 ceph-mon[29364]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 922 KiB/s wr, 54 op/s 2024-04-23T18:19:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:37 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:37 smithi131 ceph-mon[33041]: pgmap v77: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 922 KiB/s wr, 54 op/s 2024-04-23T18:19:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:37 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:37.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.75 MB/sec warmup 92 sec latency 21237.959 ms 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 -- 172.21.15.26:0/3969793909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510106170 msgr2=0x7ff51010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 --2- 172.21.15.26:0/3969793909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510106170 0x7ff51010dd40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff508009900 tx=0x7ff50802a410 comp rx=0 tx=0).stop 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 -- 172.21.15.26:0/3969793909 shutdown_connections 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 --2- 172.21.15.26:0/3969793909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff510101aa0 0x7ff510105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 --2- 172.21.15.26:0/3969793909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510106170 0x7ff51010dd40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 --2- 172.21.15.26:0/3969793909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff5101010f0 0x7ff5101014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 -- 172.21.15.26:0/3969793909 >> 172.21.15.26:0/3969793909 conn(0x7ff510077fd0 msgr2=0x7ff5100783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 -- 172.21.15.26:0/3969793909 shutdown_connections 2024-04-23T18:19:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.702+0000 7ff514a31640 1 -- 172.21.15.26:0/3969793909 wait complete. 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 Processor -- start 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 -- start start 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 0x7ff510199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff510106170 0x7ff5101999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101110c0 con 0x7ff510199f50 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff510110f40 con 0x7ff5101010f0 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff514a31640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff510111240 con 0x7ff510106170 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff50ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff50e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 0x7ff510199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff50ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42162/0 (socket says 172.21.15.26:42162) 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.703+0000 7ff50ed9d640 1 -- 172.21.15.26:0/2890871356 learned_addr learned my addr 172.21.15.26:0/2890871356 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff510106170 0x7ff5101999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 -- 172.21.15.26:0/2890871356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff510106170 msgr2=0x7ff5101999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff510106170 0x7ff5101999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 -- 172.21.15.26:0/2890871356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 msgr2=0x7ff510199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 0x7ff510199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 -- 172.21.15.26:0/2890871356 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5101a4cd0 con 0x7ff510199f50 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50e59c640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 0x7ff510199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff508004770 tx=0x7ff508004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2890871356 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50803f030 con 0x7ff510199f50 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2890871356 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff508003aa0 con 0x7ff510199f50 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5101a4f60 con 0x7ff510199f50 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2890871356 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508035070 con 0x7ff510199f50 2024-04-23T18:19:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.704+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5101a5420 con 0x7ff510199f50 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.707+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2890871356 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff508042030 con 0x7ff510199f50 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.707+0000 7ff517821740 1 Processor -- start 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.707+0000 7ff517821740 1 -- start start 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff4ff7fe640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 0x7ff4f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d865fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5588d85c0970 con 0x5588d8660350 2024-04-23T18:19:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5588d855d4a0 con 0x5588d866ac10 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff517821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5588d85bc930 con 0x5588d865fa30 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 0x7ff4f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2890871356 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff5080f0d40 con 0x7ff510199f50 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50e59c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d865fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:55780/0 (socket says 172.21.15.26:55780) 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 -- 172.21.15.26:0/4267410183 learned_addr learned my addr 172.21.15.26:0/4267410183 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 -- 172.21.15.26:0/4267410183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 msgr2=0x5588d865fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d865fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 -- 172.21.15.26:0/4267410183 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 msgr2=0x5588d866a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 -- 172.21.15.26:0/4267410183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d82d3210 con 0x5588d866ac10 2024-04-23T18:19:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50e59c640 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d865fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 0x7ff4f80a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff51006a150 tx=0x7ff504009ae0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.708+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5080b9790 tx=0x7ff5080b9190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1070b9b26fc5e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff4fcff9640 1 -- 172.21.15.26:0/4267410183 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508049020 con 0x5588d866ac10 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff4fcff9640 1 -- 172.21.15.26:0/4267410183 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff508087b60 con 0x5588d866ac10 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff4fcff9640 1 -- 172.21.15.26:0/4267410183 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5080ee740 con 0x5588d866ac10 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 -- 172.21.15.26:0/4267410183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 msgr2=0x5588d866d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5080b9790 tx=0x7ff5080b9190 comp rx=0 tx=0).stop 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 -- 172.21.15.26:0/4267410183 shutdown_connections 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866d000 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 --2- 172.21.15.26:0/4267410183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d865fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.709+0000 7ff517821740 1 -- 172.21.15.26:0/4267410183 >> 172.21.15.26:0/4267410183 conn(0x5588d865bf40 msgr2=0x5588d865c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- 172.21.15.26:0/4267410183 shutdown_connections 2024-04-23T18:19:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- 172.21.15.26:0/4267410183 wait complete. 2024-04-23T18:19:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- start start 2024-04-23T18:19:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 Processor -- start 2024-04-23T18:19:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d866a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8667e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5588d866d600 con 0x5588d8660350 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5588d855b0b0 con 0x5588d865fa30 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5588d866d780 con 0x5588d866ac10 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff517821740 10 client.0 _ll_register_callbacks cb 0x5588d8548d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42170/0 (socket says 172.21.15.26:42170) 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.710+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3228028258 learned_addr learned my addr 172.21.15.26:0/3228028258 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8667e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d866a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3228028258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 msgr2=0x5588d8667e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8667e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 msgr2=0x5588d866a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d866a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3228028258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d839f620 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8667e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c373967aa72bfe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff4f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504024070 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff5040203f0 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff508087b60 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5040205f0 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff517821740 10 client.15927 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.711+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5588d8679480 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff508087b60 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ff50402a620 con 0x5588d8660350 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 10 client.15927 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 20 client.15927 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d867a990 0x5588d867cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 4 client.15927 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588d85ce390 con 0x5588d867a990 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d867a990 0x5588d867cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff4f5ffb640 10 client.15927 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.712+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d867a990 0x5588d867cd80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5a58f717a46bbf server_cookie=c28514d9d426ab63 in_seq=0 out_seq=0 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3228028258 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588d85ce390 con 0x5588d867a990 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff4f5ffb640 10 client.15927 handle_command_reply: tid=0 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 client.15927 shutdown 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 msgr2=0x5588d8667890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50401d000 comp rx=0 tx=0).stop 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 shutdown_connections 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d867a990 0x5588d867cd80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8667e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d8667890 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 --2- 172.21.15.26:0/3228028258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d866a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff4f5ffb640 0 client.15927 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 >> 172.21.15.26:0/3228028258 conn(0x5588d865bf40 msgr2=0x5588d8656cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.713+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 shutdown_connections 2024-04-23T18:19:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.714+0000 7ff517821740 1 -- 172.21.15.26:0/3228028258 wait complete. 2024-04-23T18:19:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.714+0000 7ff517821740 20 client.15927 trim_cache size 0 max 16384 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 Processor -- start 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 -- start start 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d8667df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8672a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101ae150 con 0x5588d8660350 2024-04-23T18:19:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101ab9c0 con 0x5588d865fa30 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff517821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101a9810 con 0x5588d866ac10 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42180/0 (socket says 172.21.15.26:42180) 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3450323361 learned_addr learned my addr 172.21.15.26:0/3450323361 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d8667df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3450323361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 msgr2=0x5588d8672a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8672a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3450323361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 msgr2=0x5588d8667df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d8667df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.729+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3450323361 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d8653500 con 0x5588d8660350 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff50402ac50 tx=0x7ff504011740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c118dd7b8a8f74b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff4f7fff640 1 -- 172.21.15.26:0/3450323361 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504037030 con 0x5588d8660350 2024-04-23T18:19:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff4f7fff640 1 -- 172.21.15.26:0/3450323361 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff504009d30 con 0x5588d8660350 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff4f7fff640 1 -- 172.21.15.26:0/3450323361 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504024070 con 0x5588d8660350 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 -- 172.21.15.26:0/3450323361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 msgr2=0x5588d86763c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff50402ac50 tx=0x7ff504011740 comp rx=0 tx=0).stop 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 -- 172.21.15.26:0/3450323361 shutdown_connections 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d866ac10 0x5588d8672a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d86763c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 --2- 172.21.15.26:0/3450323361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d865fa30 0x5588d8667df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 -- 172.21.15.26:0/3450323361 >> 172.21.15.26:0/3450323361 conn(0x5588d865bf40 msgr2=0x5588d8666160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 -- 172.21.15.26:0/3450323361 shutdown_connections 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.730+0000 7ff517821740 1 -- 172.21.15.26:0/3450323361 wait complete. 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 -- start start 2024-04-23T18:19:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 Processor -- start 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d8672a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101a9e80 con 0x5588d8660350 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101ab2f0 con 0x5588d866ac10 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff5101aa550 con 0x5588d865fa30 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff517821740 10 client.0 _ll_register_callbacks cb 0x7ff50400fd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42182/0 (socket says 172.21.15.26:42182) 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3288603007 learned_addr learned my addr 172.21.15.26:0/3288603007 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d8672a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50ed9d640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3288603007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 msgr2=0x5588d8672a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d8672a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 msgr2=0x5588d866ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.731+0000 7ff50dd9b640 1 -- 172.21.15.26:0/3288603007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d866f5d0 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff50dd9b640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff504009f00 tx=0x7ff50402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3a61495eac7f2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504037030 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff5040202c0 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff504009d30 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff504024040 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 10 client.15933 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5588d86e0d40 con 0x5588d8660350 2024-04-23T18:19:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff504009d30 con 0x5588d8660350 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ff5040335e0 con 0x5588d8660350 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 10 client.15933 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 20 client.15933 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d86e13b0 0x5588d86e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 4 client.15933 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.732+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588d85c8200 con 0x5588d86e13b0 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.733+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d86e13b0 0x5588d86e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.733+0000 7ff4f5ffb640 10 client.15933 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.733+0000 7ff50e59c640 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d86e13b0 0x5588d86e37d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c911bedab3ce5965 server_cookie=e9b8675e6c36feb1 in_seq=0 out_seq=0 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.747+0000 7ff4f5ffb640 1 -- 172.21.15.26:0/3288603007 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588d85c8200 con 0x5588d86e13b0 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.747+0000 7ff4f5ffb640 10 client.15933 handle_command_reply: tid=0 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.747+0000 7ff517821740 1 client.15933 shutdown 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.747+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 msgr2=0x5588d866ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.747+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff504009f00 tx=0x7ff50402a6a0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 shutdown_connections 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5588d86e13b0 0x5588d86e37d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588d866ac10 0x5588d866ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5588d8660350 0x5588d866ea50 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff4f5ffb640 0 client.15933 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 --2- 172.21.15.26:0/3288603007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5588d865fa30 0x5588d8672a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 >> 172.21.15.26:0/3288603007 conn(0x5588d865bf40 msgr2=0x5588d86665c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 shutdown_connections 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 1 -- 172.21.15.26:0/3288603007 wait complete. 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff517821740 20 client.15933 trim_cache size 0 max 16384 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:19:37.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:19:37.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:19:37.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:19:37.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 msgr2=0x7ff4f80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 0x7ff4f80a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff51006a150 tx=0x7ff504009ae0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 msgr2=0x7ff5101a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff508004770 tx=0x7ff508004270 comp rx=0 tx=0).stop 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 shutdown_connections 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff4f80a3dc0 0x7ff4f80a6280 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff510199f50 0x7ff5101a4500 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff510106170 0x7ff5101999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 --2- 172.21.15.26:0/2890871356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5101010f0 0x7ff510199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 >> 172.21.15.26:0/2890871356 conn(0x7ff510077fd0 msgr2=0x7ff5100fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 shutdown_connections 2024-04-23T18:19:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:37.748+0000 7ff514a31640 1 -- 172.21.15.26:0/2890871356 wait complete. 2024-04-23T18:19:38.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.68 MB/sec warmup 93 sec latency 22238.100 ms 2024-04-23T18:19:39.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:39 smithi110 ceph-mon[32585]: pgmap v78: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:39 smithi026 ceph-mon[29364]: pgmap v78: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:39.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:39 smithi131 ceph-mon[33041]: pgmap v78: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:39.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.61 MB/sec warmup 94 sec latency 23238.166 ms 2024-04-23T18:19:40.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.54 MB/sec warmup 95 sec latency 24238.322 ms 2024-04-23T18:19:41.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:41 smithi110 ceph-mon[32585]: pgmap v79: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:41.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:41 smithi026 ceph-mon[29364]: pgmap v79: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:41.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:41 smithi131 ceph-mon[33041]: pgmap v79: 97 pgs: 97 active+clean; 66 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 893 KiB/s wr, 29 op/s 2024-04-23T18:19:41.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.47 MB/sec warmup 96 sec latency 25238.430 ms 2024-04-23T18:19:42.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112429 6.40 MB/sec warmup 97 sec latency 26238.478 ms 2024-04-23T18:19:42.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 -- 172.21.15.26:0/3120705139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f40103b50 msgr2=0x7f1f4010c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/3120705139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f40103b50 0x7f1f4010c030 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009900 tx=0x7f1f3002a480 comp rx=0 tx=0).stop 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 -- 172.21.15.26:0/3120705139 shutdown_connections 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/3120705139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f4010c570 0x7f1f4010e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/3120705139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f40103b50 0x7f1f4010c030 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/3120705139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 0x7f1f40103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 -- 172.21.15.26:0/3120705139 >> 172.21.15.26:0/3120705139 conn(0x7f1f40077fb0 msgr2=0x7f1f400783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 -- 172.21.15.26:0/3120705139 shutdown_connections 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.120+0000 7f1f47ee8640 1 -- 172.21.15.26:0/3120705139 wait complete. 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 Processor -- start 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 -- start start 2024-04-23T18:19:43.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 0x7f1f40198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f40103b50 0x7f1f40198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1f401121b0 con 0x7f1f4010c570 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40112030 con 0x7f1f40103b50 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f47ee8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40112330 con 0x7f1f40103230 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36448/0 (socket says 172.21.15.26:36448) 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 -- 172.21.15.26:0/1542177612 learned_addr learned my addr 172.21.15.26:0/1542177612 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f45c84640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 0x7f1f40198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f45483640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f40103b50 0x7f1f40198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 -- 172.21.15.26:0/1542177612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 msgr2=0x7f1f40198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 0x7f1f40198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 -- 172.21.15.26:0/1542177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f40103b50 msgr2=0x7f1f40198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f40103b50 0x7f1f40198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.121+0000 7f1f46485640 1 -- 172.21.15.26:0/1542177612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4019d610 con 0x7f1f4010c570 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f46485640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800cab0 tx=0x7f1f3800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3800ae20 con 0x7f1f4010c570 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f3800f900 con 0x7f1f4010c570 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4019d900 con 0x7f1f4010c570 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380189f0 con 0x7f1f4010c570 2024-04-23T18:19:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.122+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f401a5190 con 0x7f1f4010c570 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.123+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f400691c0 con 0x7f1f4010c570 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.124+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1f3800fab0 con 0x7f1f4010c570 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.125+0000 7f1f36ffd640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 0x7f1f2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.125+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f1f380d1080 con 0x7f1f4010c570 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.125+0000 7f1f45c84640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 0x7f1f2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.125+0000 7f1f45c84640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 0x7f1f2c0a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005e10 tx=0x7f1f3c005da0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.129+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f38014030 con 0x7f1f4010c570 2024-04-23T18:19:43.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.357+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f40199970 con 0x7f1f4010c570 2024-04-23T18:19:43.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.358+0000 7f1f36ffd640 1 -- 172.21.15.26:0/1542177612 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11887 (secure 0 0 0) 0x7f1f40199970 con 0x7f1f4010c570 2024-04-23T18:19:43.353 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:19:43.353 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:19:35.669505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:19:43.353 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2024-04-23T18:19:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 msgr2=0x7f1f2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 0x7f1f2c0a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005e10 tx=0x7f1f3c005da0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 msgr2=0x7f1f4019cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800cab0 tx=0x7f1f3800cf70 comp rx=0 tx=0).stop 2024-04-23T18:19:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 shutdown_connections 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f1f2c0a3d20 0x7f1f2c0a61e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f40103b50 0x7f1f40198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1f4010c570 0x7f1f4019cee0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 --2- 172.21.15.26:0/1542177612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40103230 0x7f1f40198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 >> 172.21.15.26:0/1542177612 conn(0x7f1f40077fb0 msgr2=0x7f1f400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 shutdown_connections 2024-04-23T18:19:43.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.360+0000 7f1f47ee8640 1 -- 172.21.15.26:0/1542177612 wait complete. 2024-04-23T18:19:43.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:43 smithi026 ceph-mon[29364]: pgmap v80: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-04-23T18:19:43.365 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 780d2ace --release --await' 2024-04-23T18:19:43.365 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 780d2ace --release --await 2024-04-23T18:19:43.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:43 smithi110 ceph-mon[32585]: pgmap v80: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 -- 172.21.15.26:0/677956687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc0107260 msgr2=0x7f3cc010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/677956687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc0107260 0x7f3cc010ee30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009ca0 tx=0x7f3cb802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 -- 172.21.15.26:0/677956687 shutdown_connections 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/677956687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0102b40 0x7f3cc0106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/677956687 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc0107260 0x7f3cc010ee30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/677956687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 -- 172.21.15.26:0/677956687 >> 172.21.15.26:0/677956687 conn(0x7f3cc00fd990 msgr2=0x7f3cc00ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 -- 172.21.15.26:0/677956687 shutdown_connections 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.480+0000 7f3cc7f20640 1 -- 172.21.15.26:0/677956687 wait complete. 2024-04-23T18:19:43.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 Processor -- start 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 -- start start 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0107260 0x7f3cc0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01121d0 con 0x7f3cc019cf50 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc0112050 con 0x7f3cc0107260 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc7f20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc0112350 con 0x7f3cc0102190 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36462/0 (socket says 172.21.15.26:36462) 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 -- 172.21.15.26:0/1295477574 learned_addr learned my addr 172.21.15.26:0/1295477574 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0107260 0x7f3cc0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 -- 172.21.15.26:0/1295477574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 msgr2=0x7f3cc0198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 -- 172.21.15.26:0/1295477574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0107260 msgr2=0x7f3cc0198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0107260 0x7f3cc0198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc64bd640 1 -- 172.21.15.26:0/1295477574 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc019da40 con 0x7f3cc019cf50 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.481+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cc64bd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8005cc0 tx=0x7f3cb802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cb6ffd640 1 -- 172.21.15.26:0/1295477574 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8003bf0 con 0x7f3cc019cf50 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cb6ffd640 1 -- 172.21.15.26:0/1295477574 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb8003d90 con 0x7f3cc019cf50 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cb6ffd640 1 -- 172.21.15.26:0/1295477574 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb802fbc0 con 0x7f3cc019cf50 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc019dcd0 con 0x7f3cc019cf50 2024-04-23T18:19:43.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.482+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc01a54f0 con 0x7f3cc019cf50 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.484+0000 7f3cb6ffd640 1 -- 172.21.15.26:0/1295477574 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3cb802fd60 con 0x7f3cc019cf50 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.484+0000 7f3ccac37740 1 Processor -- start 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.484+0000 7f3ccac37740 1 -- start start 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182ba750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556f1821f6d0 con 0x556f182bac90 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556f181bc1a0 con 0x556f182ba370 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3ccac37740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556f1821b690 con 0x556f182c9890 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cb6ffd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 0x7f3cac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36466/0 (socket says 172.21.15.26:36466) 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/1611312490 learned_addr learned my addr 172.21.15.26:0/1611312490 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc64bd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 0x7f3cac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cb6ffd640 1 -- 172.21.15.26:0/1295477574 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3cb80f1f00 con 0x7f3cc019cf50 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/1611312490 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 msgr2=0x556f182cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/1611312490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 msgr2=0x556f182ba750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182ba750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/1611312490 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f181c65b0 con 0x556f182bac90 2024-04-23T18:19:43.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00bd40 tx=0x7f3cbc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23aac528b2415f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.485+0000 7f3cc64bd640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 0x7f3cac0a61e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006010 tx=0x7f3cb80a8ab0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3cabfff640 1 -- 172.21.15.26:0/1611312490 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc015070 con 0x556f182bac90 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3cabfff640 1 -- 172.21.15.26:0/1611312490 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cbc00be30 con 0x556f182bac90 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3cabfff640 1 -- 172.21.15.26:0/1611312490 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0106e0 con 0x556f182bac90 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 -- 172.21.15.26:0/1611312490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 msgr2=0x556f182c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00bd40 tx=0x7f3cbc00f730 comp rx=0 tx=0).stop 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 -- 172.21.15.26:0/1611312490 shutdown_connections 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c9350 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1611312490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182ba750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 -- 172.21.15.26:0/1611312490 >> 172.21.15.26:0/1611312490 conn(0x556f182b67b0 msgr2=0x556f182b6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 -- 172.21.15.26:0/1611312490 shutdown_connections 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.486+0000 7f3ccac37740 1 -- 172.21.15.26:0/1611312490 wait complete. 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 -- start start 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 Processor -- start 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x556f182cc770 con 0x556f182c9890 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556f181b9b20 con 0x556f182bac90 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x556f182cc8f0 con 0x556f182ba370 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3ccac37740 10 client.0 _ll_register_callbacks cb 0x556f181b9530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3cc54bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3cc54bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36468/0 (socket says 172.21.15.26:36468) 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3cc54bb640 1 -- 172.21.15.26:0/1877102581 learned_addr learned my addr 172.21.15.26:0/1877102581 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.487+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 -- 172.21.15.26:0/1877102581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 msgr2=0x556f182c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182c91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 -- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 msgr2=0x556f182c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 -- 172.21.15.26:0/1877102581 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f17f31920 con 0x556f182c9890 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0012b60 tx=0x7f3cb000e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d100a761c4ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3ca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0018070 con 0x556f182c9890 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3cb0005220 con 0x556f182c9890 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cbc00be30 con 0x556f182c9890 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.488+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3cb0010ea0 con 0x556f182c9890 2024-04-23T18:19:43.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 10 client.15951 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f17ffdd30 con 0x556f182c9890 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cbc00be30 con 0x556f182c9890 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f3cb00254e0 con 0x556f182c9890 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 10 client.15951 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 10 client.15951 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 20 client.15951 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f182d9a10 0x556f182dbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 4 client.15951 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f181dd500 con 0x556f182d9a10 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.489+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f182d9a10 0x556f182dbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.490+0000 7f3ca97fa640 10 client.15951 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.490+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f182d9a10 0x556f182dbe00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d6c45622902ca4 server_cookie=9bdc828b648d8a3e in_seq=0 out_seq=0 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.490+0000 7f3ca97fa640 1 -- 172.21.15.26:0/1877102581 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f181dd500 con 0x556f182d9a10 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.490+0000 7f3ca97fa640 10 client.15951 handle_command_reply: tid=0 2024-04-23T18:19:43.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.490+0000 7f3ccac37740 1 client.15951 shutdown 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 msgr2=0x556f182c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0012b60 tx=0x7f3cb000e500 comp rx=0 tx=0).stop 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 shutdown_connections 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f182d9a10 0x556f182dbe00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182c6a40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 --2- 172.21.15.26:0/1877102581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ca97fa640 0 client.15951 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 >> 172.21.15.26:0/1877102581 conn(0x556f182b67b0 msgr2=0x556f182bc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 shutdown_connections 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 1 -- 172.21.15.26:0/1877102581 wait complete. 2024-04-23T18:19:43.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.491+0000 7f3ccac37740 20 client.15951 trim_cache size 0 max 16384 2024-04-23T18:19:43.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:43 smithi131 ceph-mon[33041]: pgmap v80: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-04-23T18:19:43.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 Processor -- start 2024-04-23T18:19:43.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 -- start start 2024-04-23T18:19:43.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182d5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01ae1c0 con 0x556f182c9890 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01aba30 con 0x556f182bac90 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3ccac37740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01a98a0 con 0x556f182ba370 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3cc54bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3cc54bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36474/0 (socket says 172.21.15.26:36474) 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.508+0000 7f3cc54bb640 1 -- 172.21.15.26:0/2572631535 learned_addr learned my addr 172.21.15.26:0/2572631535 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 -- 172.21.15.26:0/2572631535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 msgr2=0x556f182d5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182d5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 -- 172.21.15.26:0/2572631535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 msgr2=0x556f182c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 -- 172.21.15.26:0/2572631535 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f182dc170 con 0x556f182c9890 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e3a0 tx=0x7f3cb0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e049956e7cc282e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cab7fe640 1 -- 172.21.15.26:0/2572631535 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0018070 con 0x556f182c9890 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cab7fe640 1 -- 172.21.15.26:0/2572631535 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb0025ca0 con 0x556f182c9890 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.509+0000 7f3cab7fe640 1 -- 172.21.15.26:0/2572631535 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0012e60 con 0x556f182c9890 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- 172.21.15.26:0/2572631535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 msgr2=0x556f182e16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e3a0 tx=0x7f3cb0005de0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- 172.21.15.26:0/2572631535 shutdown_connections 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182bac90 0x556f182c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182c9890 0x556f182e16d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2572631535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182ba370 0x556f182d5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- 172.21.15.26:0/2572631535 >> 172.21.15.26:0/2572631535 conn(0x556f182b67b0 msgr2=0x556f182c17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- 172.21.15.26:0/2572631535 shutdown_connections 2024-04-23T18:19:43.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- 172.21.15.26:0/2572631535 wait complete. 2024-04-23T18:19:43.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 -- start start 2024-04-23T18:19:43.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.510+0000 7f3ccac37740 1 Processor -- start 2024-04-23T18:19:43.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01a9ef0 con 0x556f182bac90 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01ab360 con 0x556f182ba370 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01aa5c0 con 0x556f182c9890 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 10 client.0 _ll_register_callbacks cb 0x7f3cb00127c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:36478/0 (socket says 172.21.15.26:36478) 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/2966893105 learned_addr learned my addr 172.21.15.26:0/2966893105 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/2966893105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 msgr2=0x556f182c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 msgr2=0x556f182d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182d20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 -- 172.21.15.26:0/2966893105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f182cab30 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3cc5cbc640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00b580 tx=0x7f3cbc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2389dcaed1c7bdb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc015040 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3cbc011440 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cb0025ca0 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3cbc038050 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 10 client.15957 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.511+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f18340530 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cb0025ca0 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f3cbc004a40 con 0x556f182bac90 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 10 client.15957 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:43.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 10 client.15957 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 20 client.15957 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f18340a00 0x556f18342e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 4 client.15957 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "780d2ace", "release": true, "await": true}] 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "780d2ace", "release": true, "await": true}) -- 0x556f1822d0f0 con 0x556f18340a00 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.512+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f18340a00 0x556f18342e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.513+0000 7f3ca97fa640 10 client.15957 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:43.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.513+0000 7f3cc54bb640 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f18340a00 0x556f18342e20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4b1113ad70375f server_cookie=6fa0c8db0b43db0d in_seq=0 out_seq=0 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ca97fa640 1 -- 172.21.15.26:0/2966893105 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556f1822d0f0 con 0x556f18340a00 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ca97fa640 10 client.15957 handle_command_reply: tid=0 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 client.15957 shutdown 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 msgr2=0x556f182c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00b580 tx=0x7f3cbc00be30 comp rx=0 tx=0).stop 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 shutdown_connections 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x556f18340a00 0x556f18342e20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556f182c9890 0x556f182c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x556f182bac90 0x556f182c3be0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 --2- 172.21.15.26:0/2966893105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f182ba370 0x556f182d20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ca97fa640 0 client.15957 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 >> 172.21.15.26:0/2966893105 conn(0x556f182b67b0 msgr2=0x556f182b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.519+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 shutdown_connections 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3ccac37740 1 -- 172.21.15.26:0/2966893105 wait complete. 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3ccac37740 20 client.15957 trim_cache size 0 max 16384 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 30, 2024-04-23T18:19:43.515 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 16, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "780d2ace": { 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 16, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 86.5, 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:19:43.516 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:19:43.517 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 msgr2=0x7f3cac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 0x7f3cac0a61e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006010 tx=0x7f3cb80a8ab0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 msgr2=0x7f3cc019d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8005cc0 tx=0x7f3cb802f380 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 shutdown_connections 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3cac0a3d20 0x7f3cac0a61e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cc0107260 0x7f3cc0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3cc019cf50 0x7f3cc019d400 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 --2- 172.21.15.26:0/1295477574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3cc0102190 0x7f3cc0198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 >> 172.21.15.26:0/1295477574 conn(0x7f3cc00fd990 msgr2=0x7f3cc00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 shutdown_connections 2024-04-23T18:19:43.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:19:43.520+0000 7f3cc7f20640 1 -- 172.21.15.26:0/1295477574 wait complete. 2024-04-23T18:19:43.526 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '780d2ace', seconds elapsed: 27.6 2024-04-23T18:19:43.662 INFO:tasks.workunit.client.0.smithi026.stdout: 1 112876 6.38 MB/sec warmup 98 sec latency 27095.036 ms 2024-04-23T18:19:44.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:44 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1542177612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:44.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:44 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1542177612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:44.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:44 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1542177612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:44.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 114334 6.40 MB/sec warmup 99 sec latency 42.951 ms 2024-04-23T18:19:45.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:45 smithi110 ceph-mon[32585]: pgmap v81: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T18:19:45.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:45 smithi026 ceph-mon[29364]: pgmap v81: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T18:19:45.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:45 smithi131 ceph-mon[33041]: pgmap v81: 97 pgs: 97 active+clean; 65 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T18:19:45.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 117095 6.50 MB/sec warmup 100 sec latency 19.387 ms 2024-04-23T18:19:46.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 118515 6.51 MB/sec warmup 101 sec latency 12.890 ms 2024-04-23T18:19:47.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:47 smithi110 ceph-mon[32585]: pgmap v82: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.5 MiB/s wr, 130 op/s 2024-04-23T18:19:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:47 smithi026 ceph-mon[29364]: pgmap v82: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.5 MiB/s wr, 130 op/s 2024-04-23T18:19:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:47 smithi131 ceph-mon[33041]: pgmap v82: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.5 MiB/s wr, 130 op/s 2024-04-23T18:19:47.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 121021 6.58 MB/sec warmup 102 sec latency 9.776 ms 2024-04-23T18:19:48.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 123222 6.59 MB/sec warmup 103 sec latency 19.568 ms 2024-04-23T18:19:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:49 smithi110 ceph-mon[32585]: pgmap v83: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:49.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:49 smithi026 ceph-mon[29364]: pgmap v83: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:49.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:49 smithi131 ceph-mon[33041]: pgmap v83: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:49.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 124904 6.64 MB/sec warmup 104 sec latency 28.989 ms 2024-04-23T18:19:50.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 127801 6.74 MB/sec warmup 105 sec latency 12.913 ms 2024-04-23T18:19:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- 172.21.15.26:0/2477837919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 msgr2=0x7f190010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 --2- 172.21.15.26:0/2477837919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f190010dd40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009ca0 tx=0x7f18f802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- 172.21.15.26:0/2477837919 shutdown_connections 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 --2- 172.21.15.26:0/2477837919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1900101aa0 0x7f1900105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 --2- 172.21.15.26:0/2477837919 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f190010dd40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 --2- 172.21.15.26:0/2477837919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19001010f0 0x7f19001014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- 172.21.15.26:0/2477837919 >> 172.21.15.26:0/2477837919 conn(0x7f1900077fd0 msgr2=0x7f19000783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- 172.21.15.26:0/2477837919 shutdown_connections 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- 172.21.15.26:0/2477837919 wait complete. 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 Processor -- start 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 -- start start 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.843+0000 7f1906dd8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 0x7f19001986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f190019cf60 0x7f190019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f1900111130 con 0x7f1900106170 2024-04-23T18:19:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1900110fb0 con 0x7f19001010f0 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f19001112b0 con 0x7f190019cf60 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42286/0 (socket says 172.21.15.26:42286) 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 -- 172.21.15.26:0/3482907091 learned_addr learned my addr 172.21.15.26:0/3482907091 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1905375640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f190019cf60 0x7f190019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1904b74640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 0x7f19001986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 -- 172.21.15.26:0/3482907091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f190019cf60 msgr2=0x7f190019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f190019cf60 0x7f190019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 -- 172.21.15.26:0/3482907091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 msgr2=0x7f19001986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 0x7f19001986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 -- 172.21.15.26:0/3482907091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190019da50 con 0x7f1900106170 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1904b74640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 0x7f19001986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fffff640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00e9c0 tx=0x7f18ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fdffb640 1 -- 172.21.15.26:0/3482907091 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec00cd80 con 0x7f1900106170 2024-04-23T18:19:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190019dd40 con 0x7f1900106170 2024-04-23T18:19:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fdffb640 1 -- 172.21.15.26:0/3482907091 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18ec0054d0 con 0x7f1900106170 2024-04-23T18:19:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.844+0000 7f18fdffb640 1 -- 172.21.15.26:0/3482907091 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec010430 con 0x7f1900106170 2024-04-23T18:19:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.845+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19001a5530 con 0x7f1900106170 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.847+0000 7f18fdffb640 1 -- 172.21.15.26:0/3482907091 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f18ec004fd0 con 0x7f1900106170 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.847+0000 7f1909bc7740 1 Processor -- start 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.847+0000 7f1909bc7740 1 -- start start 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.847+0000 7f18fdffb640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 0x7f18f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 0x55d367e1d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e2c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1905375640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 0x7f18f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d367d82970 con 0x55d367e2cc10 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d367d1f4a0 con 0x55d367e1de00 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1909bc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d367d7e930 con 0x55d367e1d4e0 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fdffb640 1 -- 172.21.15.26:0/3482907091 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f18ec0d0bf0 con 0x7f1900106170 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1904b74640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 0x55d367e1d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1905375640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1904b74640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 0x55d367e1d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:34058/0 (socket says 172.21.15.26:34058) 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1905375640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42292/0 (socket says 172.21.15.26:42292) 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1904b74640 1 -- 172.21.15.26:0/1243782509 learned_addr learned my addr 172.21.15.26:0/1243782509 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e2c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 -- 172.21.15.26:0/1243782509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 msgr2=0x55d367e1d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 0x55d367e1d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 -- 172.21.15.26:0/1243782509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 msgr2=0x55d367e2f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 -- 172.21.15.26:0/1243782509 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d367a95210 con 0x55d367e1de00 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f1905375640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 0x7f18f00a6280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18f8006010 tx=0x7f18f8031bd0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18fffff640 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e2c6d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0ce370 tx=0x7f18ec098f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fed22b7335168f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.848+0000 7f18eaffd640 1 -- 172.21.15.26:0/1243782509 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec09cbc0 con 0x55d367e1de00 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f18eaffd640 1 -- 172.21.15.26:0/1243782509 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18ec0a0f70 con 0x55d367e1de00 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f18eaffd640 1 -- 172.21.15.26:0/1243782509 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0bd070 con 0x55d367e1de00 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 -- 172.21.15.26:0/1243782509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 msgr2=0x55d367e2c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e2c6d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0ce370 tx=0x7f18ec098f90 comp rx=0 tx=0).stop 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 -- 172.21.15.26:0/1243782509 shutdown_connections 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e2c6d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 --2- 172.21.15.26:0/1243782509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1d4e0 0x55d367e1d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 -- 172.21.15.26:0/1243782509 >> 172.21.15.26:0/1243782509 conn(0x55d367e19b30 msgr2=0x55d367e19f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 -- 172.21.15.26:0/1243782509 shutdown_connections 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.849+0000 7f1909bc7740 1 -- 172.21.15.26:0/1243782509 wait complete. 2024-04-23T18:19:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 -- start start 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 Processor -- start 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e2c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d367e2f720 con 0x55d367e2cc10 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d367d1d0b0 con 0x55d367e1d4e0 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d367e2f8a0 con 0x55d367e1de00 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1909bc7740 10 client.0 _ll_register_callbacks cb 0x55d367d0ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1905375640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f18fffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f1904b74640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e2c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f18fffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:34068/0 (socket says 172.21.15.26:34068) 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.850+0000 7f18fffff640 1 -- 172.21.15.26:0/4074718364 learned_addr learned my addr 172.21.15.26:0/4074718364 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 -- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 msgr2=0x55d367e2c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e2c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 -- 172.21.15.26:0/4074718364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 msgr2=0x55d367e2a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 -- 172.21.15.26:0/4074718364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d367b61620 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f1905375640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18fffff640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18ec09c8c0 tx=0x7f18ec09d380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13aff5f03bb77ede server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18dffff640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0ce5f0 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18ec0a0f70 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f18ec0a53a0 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18ec09cbc0 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f1909bc7740 10 client.25438 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d367e2f770 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.851+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18ec0a0f70 con 0x55d367e1de00 2024-04-23T18:19:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f18ec098910 con 0x55d367e1de00 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 10 client.25438 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 10 client.25438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 20 client.25438 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367e3c9c0 0x55d367e3edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 4 client.25438 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d367d90390 con 0x55d367e3c9c0 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1904b74640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367e3c9c0 0x55d367e3edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f1904b74640 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367e3c9c0 0x55d367e3edb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=865762de0cdc8384 server_cookie=60ee5b017d79be9b in_seq=0 out_seq=0 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.852+0000 7f18dffff640 10 client.25438 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f18dffff640 1 -- 172.21.15.26:0/4074718364 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d367d90390 con 0x55d367e3c9c0 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f18dffff640 10 client.25438 handle_command_reply: tid=0 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 client.25438 shutdown 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 msgr2=0x55d367e29940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18ec09c8c0 tx=0x7f18ec09d380 comp rx=0 tx=0).stop 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 shutdown_connections 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367e3c9c0 0x55d367e3edb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e2cc10 0x55d367e2a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e1de00 0x55d367e29940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4074718364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e2c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f18dffff640 0 client.25438 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.853+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 >> 172.21.15.26:0/4074718364 conn(0x55d367e19b30 msgr2=0x55d367e1f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.854+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 shutdown_connections 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.854+0000 7f1909bc7740 1 -- 172.21.15.26:0/4074718364 wait complete. 2024-04-23T18:19:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.854+0000 7f1909bc7740 20 client.25438 trim_cache size 0 max 16384 2024-04-23T18:19:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 Processor -- start 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 -- start start 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e24d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f19001ae1e0 con 0x55d367e1d4e0 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f19001aba50 con 0x55d367e1de00 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1909bc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f19001a98a0 con 0x55d367e2cc10 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1904b74640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1905375640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1904b74640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42300/0 (socket says 172.21.15.26:42300) 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f1904b74640 1 -- 172.21.15.26:0/4187934242 learned_addr learned my addr 172.21.15.26:0/4187934242 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.869+0000 7f18fffff640 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e24d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 -- 172.21.15.26:0/4187934242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 msgr2=0x55d367e2c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 -- 172.21.15.26:0/4187934242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 msgr2=0x55d367e24d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e24d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 -- 172.21.15.26:0/4187934242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d367e1dc30 con 0x55d367e1d4e0 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f1904b74640 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f18f4004700 tx=0x7f18f4014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372c968343e98868 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f18ea7fc640 1 -- 172.21.15.26:0/4187934242 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4014110 con 0x55d367e1d4e0 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f18ea7fc640 1 -- 172.21.15.26:0/4187934242 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f40142b0 con 0x55d367e1d4e0 2024-04-23T18:19:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.870+0000 7f18ea7fc640 1 -- 172.21.15.26:0/4187934242 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4005310 con 0x55d367e1d4e0 2024-04-23T18:19:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- 172.21.15.26:0/4187934242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 msgr2=0x55d367e38620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f18f4004700 tx=0x7f18f4014840 comp rx=0 tx=0).stop 2024-04-23T18:19:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- 172.21.15.26:0/4187934242 shutdown_connections 2024-04-23T18:19:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1de00 0x55d367e24d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 --2- 172.21.15.26:0/4187934242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1d4e0 0x55d367e38620 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- 172.21.15.26:0/4187934242 >> 172.21.15.26:0/4187934242 conn(0x55d367e19b30 msgr2=0x55d367e24670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- 172.21.15.26:0/4187934242 shutdown_connections 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- 172.21.15.26:0/4187934242 wait complete. 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 -- start start 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.871+0000 7f1909bc7740 1 Processor -- start 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f19001a9f10 con 0x55d367e1de00 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f19001ab380 con 0x55d367e1d4e0 2024-04-23T18:19:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f19001aa5e0 con 0x55d367e2cc10 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1909bc7740 10 client.0 _ll_register_callbacks cb 0x7f18f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18fffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18fffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42302/0 (socket says 172.21.15.26:42302) 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18fffff640 1 -- 172.21.15.26:0/2817560937 learned_addr learned my addr 172.21.15.26:0/2817560937 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:48252/0 (socket says 172.21.15.26:48252) 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1905375640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 -- 172.21.15.26:0/2817560937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 msgr2=0x55d367e2ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 -- 172.21.15.26:0/2817560937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 msgr2=0x55d367e34f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 -- 172.21.15.26:0/2817560937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d367e307d0 con 0x55d367e1d4e0 2024-04-23T18:19:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18fffff640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f1904b74640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009d60 tx=0x7f18f4009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=243221d51dce6eff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18dffff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.872+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f401e730 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d367d8a200 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f18f401e8d0 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18f4038050 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 10 client.25451 fetch_fsmap learned FSMap version 30 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d367ea2880 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d367d8a200 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f18ec0bd0c0 con 0x55d367e1d4e0 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 10 client.25451 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 10 client.25451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 20 client.25451 populate_metadata read hostname 'smithi026' 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367ea2f70 0x55d367ea5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 4 client.25451 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d367d065f0 con 0x55d367ea2f70 2024-04-23T18:19:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.873+0000 7f18fffff640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367ea2f70 0x55d367ea5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.874+0000 7f18dffff640 10 client.25451 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.874+0000 7f18fffff640 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367ea2f70 0x55d367ea5390 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c3f4f150fe235a server_cookie=a84bb24ce221ab14 in_seq=0 out_seq=0 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f18dffff640 1 -- 172.21.15.26:0/2817560937 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d367d065f0 con 0x55d367ea2f70 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f18dffff640 10 client.25451 handle_command_reply: tid=0 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 client.25451 shutdown 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 msgr2=0x55d367e34a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009d60 tx=0x7f18f4009ee0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 shutdown_connections 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d367ea2f70 0x55d367ea5390 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d367e2cc10 0x55d367e2ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d367e1de00 0x55d367e34f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 --2- 172.21.15.26:0/2817560937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d367e1d4e0 0x55d367e34a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f18dffff640 0 client.25451 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 >> 172.21.15.26:0/2817560937 conn(0x55d367e19b30 msgr2=0x55d367e24f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 shutdown_connections 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 1 -- 172.21.15.26:0/2817560937 wait complete. 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.909+0000 7f1909bc7740 20 client.25451 trim_cache size 0 max 16384 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:19:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:19:50.905 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 msgr2=0x7f18f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.905 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 0x7f18f00a6280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18f8006010 tx=0x7f18f8031bd0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.905 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 msgr2=0x7f1900198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00e9c0 tx=0x7f18ec00ee80 comp rx=0 tx=0).stop 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 shutdown_connections 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f18f00a3dc0 0x7f18f00a6280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f190019cf60 0x7f190019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1900106170 0x7f1900198be0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 --2- 172.21.15.26:0/3482907091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19001010f0 0x7f19001986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 >> 172.21.15.26:0/3482907091 conn(0x7f1900077fd0 msgr2=0x7f1900104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 shutdown_connections 2024-04-23T18:19:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:19:50.910+0000 7f1906dd8640 1 -- 172.21.15.26:0/3482907091 wait complete. 2024-04-23T18:19:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:51 smithi110 ceph-mon[32585]: pgmap v84: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:51.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:51 smithi026 ceph-mon[29364]: pgmap v84: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:51 smithi131 ceph-mon[33041]: pgmap v84: 97 pgs: 97 active+clean; 64 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 1.8 MiB/s wr, 107 op/s 2024-04-23T18:19:51.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 129180 6.74 MB/sec warmup 106 sec latency 22.746 ms 2024-04-23T18:19:52.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 131665 6.81 MB/sec warmup 107 sec latency 11.342 ms 2024-04-23T18:19:53.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:53 smithi110 ceph-mon[32585]: pgmap v85: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 229 op/s 2024-04-23T18:19:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:53 smithi026 ceph-mon[29364]: pgmap v85: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 229 op/s 2024-04-23T18:19:53.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:53 smithi131 ceph-mon[33041]: pgmap v85: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 229 op/s 2024-04-23T18:19:53.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 133873 6.82 MB/sec warmup 108 sec latency 21.520 ms 2024-04-23T18:19:54.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 135695 6.86 MB/sec warmup 109 sec latency 41.167 ms 2024-04-23T18:19:55.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:55 smithi110 ceph-mon[32585]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-23T18:19:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:55 smithi026 ceph-mon[29364]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-23T18:19:55.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:55 smithi131 ceph-mon[33041]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-23T18:19:55.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 138585 6.97 MB/sec warmup 110 sec latency 13.097 ms 2024-04-23T18:19:56.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 140060 6.96 MB/sec warmup 111 sec latency 13.626 ms 2024-04-23T18:19:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:57 smithi110 ceph-mon[32585]: pgmap v87: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-04-23T18:19:57.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:57 smithi026 ceph-mon[29364]: pgmap v87: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-04-23T18:19:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:57 smithi131 ceph-mon[33041]: pgmap v87: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-04-23T18:19:57.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 142380 7.02 MB/sec warmup 112 sec latency 8.312 ms 2024-04-23T18:19:58.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 144687 7.05 MB/sec warmup 113 sec latency 14.568 ms 2024-04-23T18:19:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:19:59 smithi110 ceph-mon[32585]: pgmap v88: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:19:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:19:59 smithi026 ceph-mon[29364]: pgmap v88: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:19:59.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:19:59 smithi131 ceph-mon[33041]: pgmap v88: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:19:59.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 146250 7.07 MB/sec warmup 114 sec latency 13.902 ms 2024-04-23T18:20:00.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:00 smithi110 ceph-mon[32585]: overall HEALTH_OK 2024-04-23T18:20:00.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:00 smithi026 ceph-mon[29364]: overall HEALTH_OK 2024-04-23T18:20:00.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:00 smithi131 ceph-mon[33041]: overall HEALTH_OK 2024-04-23T18:20:00.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 147286 7.05 MB/sec warmup 115 sec latency 525.413 ms 2024-04-23T18:20:01.359 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:01 smithi026 ceph-mon[29364]: pgmap v89: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:20:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:01 smithi110 ceph-mon[32585]: pgmap v89: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:20:01.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:01 smithi131 ceph-mon[33041]: pgmap v89: 97 pgs: 97 active+clean; 75 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 252 op/s 2024-04-23T18:20:01.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 149246 7.10 MB/sec warmup 116 sec latency 540.111 ms 2024-04-23T18:20:02.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 150609 7.09 MB/sec warmup 117 sec latency 14.215 ms 2024-04-23T18:20:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:03 smithi110 ceph-mon[32585]: pgmap v90: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.1 MiB/s wr, 356 op/s 2024-04-23T18:20:03.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:03 smithi026 ceph-mon[29364]: pgmap v90: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.1 MiB/s wr, 356 op/s 2024-04-23T18:20:03.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:03 smithi131 ceph-mon[33041]: pgmap v90: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.1 MiB/s wr, 356 op/s 2024-04-23T18:20:03.663 INFO:tasks.workunit.client.0.smithi026.stdout: 1 152939 7.15 MB/sec warmup 118 sec latency 18.169 ms 2024-04-23T18:20:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- 172.21.15.26:0/905328336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a8107270 msgr2=0x7f48a810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/905328336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a8107270 0x7f48a810ee40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009ca0 tx=0x7f48a002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:20:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- 172.21.15.26:0/905328336 shutdown_connections 2024-04-23T18:20:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/905328336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a8107270 0x7f48a810ee40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/905328336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8102b50 0x7f48a8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/905328336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- 172.21.15.26:0/905328336 >> 172.21.15.26:0/905328336 conn(0x7f48a80fda00 msgr2=0x7f48a80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- 172.21.15.26:0/905328336 shutdown_connections 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- 172.21.15.26:0/905328336 wait complete. 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 Processor -- start 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.003+0000 7f48aeb6f640 1 -- start start 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8107270 0x7f48a8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81121e0 con 0x7f48a819cf50 2024-04-23T18:20:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8112060 con 0x7f48a81021a0 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8112360 con 0x7f48a8107270 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43440/0 (socket says 172.21.15.26:43440) 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8107270 0x7f48a8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 -- 172.21.15.26:0/3230594204 learned_addr learned my addr 172.21.15.26:0/3230594204 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ac90b640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 -- 172.21.15.26:0/3230594204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8107270 msgr2=0x7f48a8198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8107270 0x7f48a8198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 -- 172.21.15.26:0/3230594204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 msgr2=0x7f48a81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 -- 172.21.15.26:0/3230594204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a819da40 con 0x7f48a819cf50 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ac90b640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48ad10c640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f48a0005cc0 tx=0x7f48a0031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48a5ffb640 1 -- 172.21.15.26:0/3230594204 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a0003b70 con 0x7f48a819cf50 2024-04-23T18:20:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48a5ffb640 1 -- 172.21.15.26:0/3230594204 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48a00318f0 con 0x7f48a819cf50 2024-04-23T18:20:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a819dcd0 con 0x7f48a819cf50 2024-04-23T18:20:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.004+0000 7f48a5ffb640 1 -- 172.21.15.26:0/3230594204 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a002f950 con 0x7f48a819cf50 2024-04-23T18:20:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.005+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a81a5520 con 0x7f48a819cf50 2024-04-23T18:20:04.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.007+0000 7f48a5ffb640 1 -- 172.21.15.26:0/3230594204 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48a002fb50 con 0x7f48a819cf50 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.007+0000 7f48b1964740 1 Processor -- start 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.007+0000 7f48b1964740 1 -- start start 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3c3c10 0x55b72c3c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b72c319970 con 0x55b72c3b4e60 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b72c2b64a0 con 0x55b72c3c3c10 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48b1964740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b72c315930 con 0x55b72c3b4540 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a5ffb640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 0x7f48900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48ac90b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:59320/0 (socket says 172.21.15.26:59320) 2024-04-23T18:20:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 -- 172.21.15.26:0/323703161 learned_addr learned my addr 172.21.15.26:0/323703161 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3c3c10 0x55b72c3c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48ad10c640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 0x7f48900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a5ffb640 1 -- 172.21.15.26:0/3230594204 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f48a00f19c0 con 0x7f48a819cf50 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 -- 172.21.15.26:0/323703161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3c3c10 msgr2=0x55b72c3c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3c3c10 0x55b72c3c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 -- 172.21.15.26:0/323703161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 msgr2=0x55b72c3c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 -- 172.21.15.26:0/323703161 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72c02c210 con 0x55b72c3b4540 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48a7fff640 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f489c009fd0 tx=0x7f489c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39abe165a03df8c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.008+0000 7f48ad10c640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 0x7f48900a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48a00bd740 tx=0x7f48a002a8c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f488effd640 1 -- 172.21.15.26:0/323703161 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c004050 con 0x55b72c3b4540 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f488effd640 1 -- 172.21.15.26:0/323703161 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c00fc70 con 0x55b72c3b4540 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f488effd640 1 -- 172.21.15.26:0/323703161 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c003950 con 0x55b72c3b4540 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 -- 172.21.15.26:0/323703161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 msgr2=0x55b72c3b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f489c009fd0 tx=0x7f489c004730 comp rx=0 tx=0).stop 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 -- 172.21.15.26:0/323703161 shutdown_connections 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3c3c10 0x55b72c3c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 --2- 172.21.15.26:0/323703161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3b4920 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 -- 172.21.15.26:0/323703161 >> 172.21.15.26:0/323703161 conn(0x55b72c3b0b30 msgr2=0x55b72c3b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 -- 172.21.15.26:0/323703161 shutdown_connections 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.009+0000 7f48b1964740 1 -- 172.21.15.26:0/323703161 wait complete. 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 -- start start 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 Processor -- start 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b72c3c6840 con 0x55b72c3b4e60 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b72c2b40b0 con 0x55b72c3b4540 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b72c3c69c0 con 0x55b72c3c3c10 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48b1964740 10 client.0 _ll_register_callbacks cb 0x55b72c2a1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48ac90b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:52768/0 (socket says 172.21.15.26:52768) 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48a7fff640 1 -- 172.21.15.26:0/2139193828 learned_addr learned my addr 172.21.15.26:0/2139193828 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.010+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 -- 172.21.15.26:0/2139193828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 msgr2=0x55b72c3c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 -- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 msgr2=0x55b72c3c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 -- 172.21.15.26:0/2139193828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72c0f8620 con 0x55b72c3b4e60 2024-04-23T18:20:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48ac90b640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c0870 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4898012cd0 tx=0x7f489800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695d9683d91480d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f4887fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898012110 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f489c00fc70 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4898004050 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f48980122b0 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48b1964740 10 client.15987 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b72c3d27f0 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.011+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c00fc70 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f4898010430 con 0x55b72c3b4e60 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 10 client.15987 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 20 client.15987 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c3d3d00 0x55b72c3d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 4 client.15987 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b72c327390 con 0x55b72c3d3d00 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c3d3d00 0x55b72c3d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c3d3d00 0x55b72c3d60f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efc4cbef834249cf server_cookie=1b09d7e1da516ab7 in_seq=0 out_seq=0 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.012+0000 7f4887fff640 10 client.15987 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f4887fff640 1 -- 172.21.15.26:0/2139193828 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b72c327390 con 0x55b72c3d3d00 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f4887fff640 10 client.15987 handle_command_reply: tid=0 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 client.15987 shutdown 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 msgr2=0x55b72c3c0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c0870 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4898012cd0 tx=0x7f489800f790 comp rx=0 tx=0).stop 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 shutdown_connections 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c3d3d00 0x55b72c3d60f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4e60 0x55b72c3c0870 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 --2- 172.21.15.26:0/2139193828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3b4540 0x55b72c3c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f4887fff640 0 client.15987 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 >> 172.21.15.26:0/2139193828 conn(0x55b72c3b0b30 msgr2=0x55b72c3b69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 shutdown_connections 2024-04-23T18:20:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 1 -- 172.21.15.26:0/2139193828 wait complete. 2024-04-23T18:20:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.013+0000 7f48b1964740 20 client.15987 trim_cache size 0 max 16384 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.031+0000 7f48b1964740 1 Processor -- start 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.031+0000 7f48b1964740 1 -- start start 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3db9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81ae140 con 0x55b72c3b4540 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81ab9b0 con 0x55b72c3bf5a0 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48b1964740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81a9800 con 0x55b72c3c3c10 2024-04-23T18:20:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:43472/0 (socket says 172.21.15.26:43472) 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 -- 172.21.15.26:0/2335416977 learned_addr learned my addr 172.21.15.26:0/2335416977 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3db9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48ac90b640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 -- 172.21.15.26:0/2335416977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 msgr2=0x55b72c3afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 -- 172.21.15.26:0/2335416977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 msgr2=0x55b72c3db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3db9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 -- 172.21.15.26:0/2335416977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b72c3d6750 con 0x55b72c3b4540 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48ac90b640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f48a7fff640 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f489c009fd0 tx=0x7f489c0044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd715e7d8622fc69 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.032+0000 7f488e7fc640 1 -- 172.21.15.26:0/2335416977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c00fbf0 con 0x55b72c3b4540 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f488e7fc640 1 -- 172.21.15.26:0/2335416977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c030880 con 0x55b72c3b4540 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f488e7fc640 1 -- 172.21.15.26:0/2335416977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c0053a0 con 0x55b72c3b4540 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 -- 172.21.15.26:0/2335416977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 msgr2=0x55b72c3c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f489c009fd0 tx=0x7f489c0044e0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 -- 172.21.15.26:0/2335416977 shutdown_connections 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3db9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3c3c10 0x55b72c3afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 --2- 172.21.15.26:0/2335416977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3b4540 0x55b72c3c30b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 -- 172.21.15.26:0/2335416977 >> 172.21.15.26:0/2335416977 conn(0x55b72c3b0b30 msgr2=0x55b72c3af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 -- 172.21.15.26:0/2335416977 shutdown_connections 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.033+0000 7f48b1964740 1 -- 172.21.15.26:0/2335416977 wait complete. 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 -- start start 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 Processor -- start 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 0x55b72c433290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81a9e70 con 0x55b72c3c3c10 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81ab2e0 con 0x55b72c3bf5a0 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f48a81aa540 con 0x55b72c3b4540 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48b1964740 10 client.0 _ll_register_callbacks cb 0x7f4898012910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48ac90b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 0x55b72c433290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48ac90b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:52778/0 (socket says 172.21.15.26:52778) 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.034+0000 7f48ac90b640 1 -- 172.21.15.26:0/1252043578 learned_addr learned my addr 172.21.15.26:0/1252043578 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 -- 172.21.15.26:0/1252043578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 msgr2=0x55b72c3c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 -- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 msgr2=0x55b72c3c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 -- 172.21.15.26:0/1252043578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72c433850 con 0x55b72c3c3c10 2024-04-23T18:20:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48a7fff640 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 0x55b72c433290 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f489c011710 tx=0x7f489c004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba11171f07c85802 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c005050 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f489c0051f0 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f489c030880 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f489c0053f0 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48b1964740 10 client.15993 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b72c43ab80 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c030880 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f489c03b020 con 0x55b72c3c3c10 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.035+0000 7f48b1964740 10 client.15993 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48b1964740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48b1964740 20 client.15993 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c43b050 0x55b72c43d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48b1964740 4 client.15993 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b72c321200 con 0x55b72c43b050 2024-04-23T18:20:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48ac90b640 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c43b050 0x55b72c43d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f4887fff640 10 client.15993 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.036+0000 7f48ac90b640 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c43b050 0x55b72c43d440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=981d2182e094e57 server_cookie=5fbbdb7b4d9f0cff in_seq=0 out_seq=0 2024-04-23T18:20:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f4887fff640 1 -- 172.21.15.26:0/1252043578 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b72c321200 con 0x55b72c43b050 2024-04-23T18:20:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f4887fff640 10 client.15993 handle_command_reply: tid=0 2024-04-23T18:20:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 client.15993 shutdown 2024-04-23T18:20:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 msgr2=0x55b72c433290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 0x55b72c433290 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f489c011710 tx=0x7f489c004550 comp rx=0 tx=0).stop 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 shutdown_connections 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b72c43b050 0x55b72c43d440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b72c3bf5a0 0x55b72c3c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f4887fff640 0 client.15993 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b72c3c3c10 0x55b72c433290 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 --2- 172.21.15.26:0/1252043578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b72c3b4540 0x55b72c3c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 >> 172.21.15.26:0/1252043578 conn(0x55b72c3b0b30 msgr2=0x55b72c3b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 shutdown_connections 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 1 -- 172.21.15.26:0/1252043578 wait complete. 2024-04-23T18:20:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.087+0000 7f48b1964740 20 client.15993 trim_cache size 0 max 16384 2024-04-23T18:20:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:20:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:20:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:20:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 msgr2=0x7f48900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 0x7f48900a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48a00bd740 tx=0x7f48a002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 msgr2=0x7f48a819d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f48a0005cc0 tx=0x7f48a0031700 comp rx=0 tx=0).stop 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 shutdown_connections 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f48900a3e40 0x7f48900a6300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f48a819cf50 0x7f48a819d400 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a8107270 0x7f48a8198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 --2- 172.21.15.26:0/3230594204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48a81021a0 0x7f48a81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 >> 172.21.15.26:0/3230594204 conn(0x7f48a80fda00 msgr2=0x7f48a80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 shutdown_connections 2024-04-23T18:20:04.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:04.088+0000 7f48aeb6f640 1 -- 172.21.15.26:0/3230594204 wait complete. 2024-04-23T18:20:04.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 154885 7.14 MB/sec warmup 119 sec latency 29.690 ms 2024-04-23T18:20:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:05 smithi110 ceph-mon[32585]: pgmap v91: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2024-04-23T18:20:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:05 smithi026 ceph-mon[29364]: pgmap v91: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2024-04-23T18:20:05.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:05 smithi131 ceph-mon[33041]: pgmap v91: 97 pgs: 97 active+clean; 80 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2024-04-23T18:20:06.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 159455 14.47 MB/sec execute 1 sec latency 15.189 ms 2024-04-23T18:20:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:07 smithi110 ceph-mon[32585]: pgmap v92: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 388 op/s 2024-04-23T18:20:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:07 smithi026 ceph-mon[29364]: pgmap v92: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 388 op/s 2024-04-23T18:20:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:07 smithi131 ceph-mon[33041]: pgmap v92: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 388 op/s 2024-04-23T18:20:07.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 160877 10.61 MB/sec execute 2 sec latency 13.196 ms 2024-04-23T18:20:08.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 163507 12.64 MB/sec execute 3 sec latency 14.358 ms 2024-04-23T18:20:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:09 smithi110 ceph-mon[32585]: pgmap v93: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:09 smithi026 ceph-mon[29364]: pgmap v93: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:09.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:09 smithi131 ceph-mon[33041]: pgmap v93: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:09.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 165590 11.32 MB/sec execute 4 sec latency 14.433 ms 2024-04-23T18:20:10.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 167399 11.49 MB/sec execute 5 sec latency 12.088 ms 2024-04-23T18:20:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:11 smithi110 ceph-mon[32585]: pgmap v94: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:11 smithi026 ceph-mon[29364]: pgmap v94: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:11.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:11 smithi131 ceph-mon[33041]: pgmap v94: 97 pgs: 97 active+clean; 86 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 258 op/s 2024-04-23T18:20:11.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 170347 12.50 MB/sec execute 6 sec latency 10.018 ms 2024-04-23T18:20:12.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 171721 11.65 MB/sec execute 7 sec latency 15.865 ms 2024-04-23T18:20:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:13 smithi110 ceph-mon[32585]: pgmap v95: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 368 op/s 2024-04-23T18:20:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:13 smithi026 ceph-mon[29364]: pgmap v95: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 368 op/s 2024-04-23T18:20:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:13 smithi131 ceph-mon[33041]: pgmap v95: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 368 op/s 2024-04-23T18:20:13.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 174226 11.95 MB/sec execute 8 sec latency 11.940 ms 2024-04-23T18:20:14.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 176307 11.48 MB/sec execute 9 sec latency 19.204 ms 2024-04-23T18:20:15.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:15 smithi110 ceph-mon[32585]: pgmap v96: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 263 op/s 2024-04-23T18:20:15.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:15 smithi026 ceph-mon[29364]: pgmap v96: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 263 op/s 2024-04-23T18:20:15.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:15 smithi131 ceph-mon[33041]: pgmap v96: 97 pgs: 97 active+clean; 93 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 263 op/s 2024-04-23T18:20:15.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 178146 11.51 MB/sec execute 10 sec latency 14.913 ms 2024-04-23T18:20:16.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 181007 12.03 MB/sec execute 11 sec latency 11.021 ms 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 -- 172.21.15.26:0/423848473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee58101aa0 msgr2=0x7fee58105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 --2- 172.21.15.26:0/423848473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee58101aa0 0x7fee58105a90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fee48009900 tx=0x7fee4802a410 comp rx=0 tx=0).stop 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 -- 172.21.15.26:0/423848473 shutdown_connections 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 --2- 172.21.15.26:0/423848473 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 0x7fee5810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 --2- 172.21.15.26:0/423848473 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee58101aa0 0x7fee58105a90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 --2- 172.21.15.26:0/423848473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 0x7fee581014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 -- 172.21.15.26:0/423848473 >> 172.21.15.26:0/423848473 conn(0x7fee58077fd0 msgr2=0x7fee580783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 -- 172.21.15.26:0/423848473 shutdown_connections 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.180+0000 7fee5fa31640 1 -- 172.21.15.26:0/423848473 wait complete. 2024-04-23T18:20:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 Processor -- start 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 -- start start 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 0x7fee58102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 0x7fee58103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fee58111090 con 0x7fee581035b0 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fee58110f10 con 0x7fee581010f0 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5fa31640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fee58111210 con 0x7fee58106170 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53144/0 (socket says 172.21.15.26:53144) 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 -- 172.21.15.26:0/959892166 learned_addr learned my addr 172.21.15.26:0/959892166 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 0x7fee58102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 0x7fee58103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 -- 172.21.15.26:0/959892166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 msgr2=0x7fee58103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 0x7fee58103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 -- 172.21.15.26:0/959892166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 msgr2=0x7fee58102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 0x7fee58102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 -- 172.21.15.26:0/959892166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee581a6f60 con 0x7fee581035b0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.181+0000 7fee5dfce640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fee5000eb10 tx=0x7fee5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.182+0000 7fee4e7fc640 1 -- 172.21.15.26:0/959892166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50004b20 con 0x7fee581035b0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.182+0000 7fee4e7fc640 1 -- 172.21.15.26:0/959892166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee50004cc0 con 0x7fee581035b0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.182+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee581a7250 con 0x7fee581035b0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.182+0000 7fee4e7fc640 1 -- 172.21.15.26:0/959892166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50010670 con 0x7fee581035b0 2024-04-23T18:20:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.182+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee581a7720 con 0x7fee581035b0 2024-04-23T18:20:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.184+0000 7fee4e7fc640 1 -- 172.21.15.26:0/959892166 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee50005020 con 0x7fee581035b0 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.184+0000 7fee62823740 1 Processor -- start 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.184+0000 7fee62823740 1 -- start start 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.184+0000 7fee62823740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d23a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.184+0000 7fee62823740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee62823740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d30bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee62823740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x561630c88970 con 0x561630d23f40 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee62823740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561630c254a0 con 0x561630d2e800 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee62823740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x561630c84930 con 0x561630d23620 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee4e7fc640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 0x7fee440a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d23a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53150/0 (socket says 172.21.15.26:53150) 2024-04-23T18:20:17.185 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5d7cd640 1 -- 172.21.15.26:0/1569013925 learned_addr learned my addr 172.21.15.26:0/1569013925 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d30bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5dfce640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 0x7fee440a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee4e7fc640 1 -- 172.21.15.26:0/959892166 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fee500d0d30 con 0x7fee581035b0 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1569013925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 msgr2=0x561630d30bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d30bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1569013925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 msgr2=0x561630d2e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1569013925 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56163099b210 con 0x561630d23620 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d23a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee48006010 tx=0x7fee48005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21af1aaed745a7ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee5dfce640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 0x7fee440a6280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fee5009c9a0 tx=0x7fee5000efd0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee437fe640 1 -- 172.21.15.26:0/1569013925 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48035030 con 0x561630d23620 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee437fe640 1 -- 172.21.15.26:0/1569013925 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee480317e0 con 0x561630d23620 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.185+0000 7fee437fe640 1 -- 172.21.15.26:0/1569013925 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48003cb0 con 0x561630d23620 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 -- 172.21.15.26:0/1569013925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 msgr2=0x561630d23a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d23a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee48006010 tx=0x7fee48005780 comp rx=0 tx=0).stop 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 -- 172.21.15.26:0/1569013925 shutdown_connections 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d30bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 --2- 172.21.15.26:0/1569013925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d23a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 -- 172.21.15.26:0/1569013925 >> 172.21.15.26:0/1569013925 conn(0x561630d1fb30 msgr2=0x561630d1ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 -- 172.21.15.26:0/1569013925 shutdown_connections 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.186+0000 7fee62823740 1 -- 172.21.15.26:0/1569013925 wait complete. 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 -- start start 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 Processor -- start 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23620 0x561630d2e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x561630d313d0 con 0x561630d2e800 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561630c230b0 con 0x561630d23620 2024-04-23T18:20:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x561630d31550 con 0x561630d23f40 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee62823740 10 client.0 _ll_register_callbacks cb 0x561630c10d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:55094/0 (socket says 172.21.15.26:55094) 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53160/0 (socket says 172.21.15.26:53160) 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5d7cd640 1 -- 172.21.15.26:0/2334037807 learned_addr learned my addr 172.21.15.26:0/2334037807 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23620 0x561630d2e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5d7cd640 1 -- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23620 msgr2=0x561630d2e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.187+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23620 0x561630d2e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee5d7cd640 1 -- 172.21.15.26:0/2334037807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 msgr2=0x561630d2bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee5d7cd640 1 -- 172.21.15.26:0/2334037807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561630a67620 con 0x561630d23f40 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee54012840 tx=0x7fee54012d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972d11f9c33c86e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54004050 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee480317e0 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fee540041f0 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee540043f0 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 10 client.25471 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561630d3d430 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee480317e0 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fee54003a90 con 0x561630d23f40 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 10 client.25471 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 10 client.25471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 20 client.25471 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630d3e8a0 0x561630d40c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 4 client.25471 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.188+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561630c96390 con 0x561630d3e8a0 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.189+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630d3e8a0 0x561630d40c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.189+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630d3e8a0 0x561630d40c90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e75006a4d08bd874 server_cookie=5cc223d7a8774a2f in_seq=0 out_seq=0 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.189+0000 7fee40ff9640 10 client.25471 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee40ff9640 1 -- 172.21.15.26:0/2334037807 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561630c96390 con 0x561630d3e8a0 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee40ff9640 10 client.25471 handle_command_reply: tid=0 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 client.25471 shutdown 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 msgr2=0x561630d2b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee54012840 tx=0x7fee54012d00 comp rx=0 tx=0).stop 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 shutdown_connections 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630d3e8a0 0x561630d40c90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d2e800 0x561630d2bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee40ff9640 0 client.25471 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23f40 0x561630d2b4f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 --2- 172.21.15.26:0/2334037807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23620 0x561630d2e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 >> 172.21.15.26:0/2334037807 conn(0x561630d1fb30 msgr2=0x561630d258b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 shutdown_connections 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 1 -- 172.21.15.26:0/2334037807 wait complete. 2024-04-23T18:20:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.190+0000 7fee62823740 20 client.25471 trim_cache size 0 max 16384 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.205+0000 7fee62823740 1 Processor -- start 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.205+0000 7fee62823740 1 -- start start 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.205+0000 7fee62823740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee62823740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee62823740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d46560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee62823740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fee581b2570 con 0x561630d23f40 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee62823740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fee581afde0 con 0x561630d2e800 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee62823740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fee581adc50 con 0x561630d23620 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:55106/0 (socket says 172.21.15.26:55106) 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1549575124 learned_addr learned my addr 172.21.15.26:0/1549575124 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5d7cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53176/0 (socket says 172.21.15.26:53176) 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d46560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1549575124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 msgr2=0x561630d46560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d46560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1549575124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 msgr2=0x561630d2dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 -- 172.21.15.26:0/1549575124 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561630d41000 con 0x561630d23620 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fee48000c00 tx=0x7fee48032d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=364975a2486ee375 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee42ffd640 1 -- 172.21.15.26:0/1549575124 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48035040 con 0x561630d23620 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.206+0000 7fee42ffd640 1 -- 172.21.15.26:0/1549575124 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee48031780 con 0x561630d23620 2024-04-23T18:20:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee42ffd640 1 -- 172.21.15.26:0/1549575124 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48004330 con 0x561630d23620 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 -- 172.21.15.26:0/1549575124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 msgr2=0x561630d3a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fee48000c00 tx=0x7fee48032d60 comp rx=0 tx=0).stop 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 -- 172.21.15.26:0/1549575124 shutdown_connections 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d2e800 0x561630d46560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23f40 0x561630d2dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 --2- 172.21.15.26:0/1549575124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d23620 0x561630d3a290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 -- 172.21.15.26:0/1549575124 >> 172.21.15.26:0/1549575124 conn(0x561630d1fb30 msgr2=0x561630d24fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 -- 172.21.15.26:0/1549575124 shutdown_connections 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.207+0000 7fee62823740 1 -- 172.21.15.26:0/1549575124 wait complete. 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 -- start start 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 Processor -- start 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 0x561630d287c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d2e800 0x561630d28d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fee581ae2a0 con 0x561630d23620 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fee581af710 con 0x561630d23f40 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fee581ae970 con 0x561630d2e800 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee62823740 10 client.0 _ll_register_callbacks cb 0x7fee54012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee5cfcc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53186/0 (socket says 172.21.15.26:53186) 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee5cfcc640 1 -- 172.21.15.26:0/2699288292 learned_addr learned my addr 172.21.15.26:0/2699288292 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 0x561630d287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.208+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d2e800 0x561630d28d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 -- 172.21.15.26:0/2699288292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d2e800 msgr2=0x561630d28d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d2e800 0x561630d28d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 -- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 msgr2=0x561630d287c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 0x561630d287c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 -- 172.21.15.26:0/2699288292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561630da3e80 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 0x561630d287c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee5cfcc640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fee4802a930 tx=0x7fee4802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d5a174215f5f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48035070 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fee48031d30 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee48031780 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee48056330 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee62823740 10 client.16011 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561630da5170 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.209+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee48031780 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fee48004750 con 0x561630d23620 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 10 client.16011 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 10 client.16011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 20 client.16011 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630da5390 0x561630da7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 4 client.16011 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561630c90200 con 0x561630da5390 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630da5390 0x561630da7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee5d7cd640 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630da5390 0x561630da7780 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b4235d022c2943 server_cookie=47504cb534781b11 in_seq=0 out_seq=0 2024-04-23T18:20:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.210+0000 7fee40ff9640 10 client.16011 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee40ff9640 1 -- 172.21.15.26:0/2699288292 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561630c90200 con 0x561630da5390 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee40ff9640 10 client.16011 handle_command_reply: tid=0 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 client.16011 shutdown 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 msgr2=0x561630d28280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fee4802a930 tx=0x7fee4802a410 comp rx=0 tx=0).stop 2024-04-23T18:20:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 shutdown_connections 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x561630da5390 0x561630da7780 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561630d23f40 0x561630d287c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561630d2e800 0x561630d28d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 --2- 172.21.15.26:0/2699288292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x561630d23620 0x561630d28280 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee40ff9640 0 client.16011 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 >> 172.21.15.26:0/2699288292 conn(0x561630d1fb30 msgr2=0x561630d24fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 shutdown_connections 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.259+0000 7fee62823740 1 -- 172.21.15.26:0/2699288292 wait complete. 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee62823740 20 client.16011 trim_cache size 0 max 16384 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:20:17.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 msgr2=0x7fee440a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 0x7fee440a6280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fee5009c9a0 tx=0x7fee5000efd0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 msgr2=0x7fee581a68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fee5000eb10 tx=0x7fee5000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 shutdown_connections 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fee440a3dc0 0x7fee440a6280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fee581035b0 0x7fee581a68c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee58106170 0x7fee58103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 --2- 172.21.15.26:0/959892166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee581010f0 0x7fee58102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 >> 172.21.15.26:0/959892166 conn(0x7fee58077fd0 msgr2=0x7fee580699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 shutdown_connections 2024-04-23T18:20:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:17.260+0000 7fee5fa31640 1 -- 172.21.15.26:0/959892166 wait complete. 2024-04-23T18:20:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:17 smithi110 ceph-mon[32585]: pgmap v97: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 378 op/s 2024-04-23T18:20:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:17 smithi026 ceph-mon[29364]: pgmap v97: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 378 op/s 2024-04-23T18:20:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:17 smithi131 ceph-mon[33041]: pgmap v97: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 378 op/s 2024-04-23T18:20:17.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 182315 11.57 MB/sec execute 12 sec latency 16.460 ms 2024-04-23T18:20:18.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:18 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:18 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:18 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:18.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:18 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:20:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:18 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:18 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:18 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:18.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:18 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:20:18.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:18 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:18.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:18 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:18.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:18 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:18.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:18 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:20:18.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 184814 11.76 MB/sec execute 13 sec latency 12.356 ms 2024-04-23T18:20:19.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:19 smithi110 ceph-mon[32585]: pgmap v98: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:19 smithi026 ceph-mon[29364]: pgmap v98: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:19.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:19 smithi131 ceph-mon[33041]: pgmap v98: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:19.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 186891 11.43 MB/sec execute 14 sec latency 17.398 ms 2024-04-23T18:20:20.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 188598 11.47 MB/sec execute 15 sec latency 28.258 ms 2024-04-23T18:20:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:21 smithi110 ceph-mon[32585]: pgmap v99: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:21 smithi026 ceph-mon[29364]: pgmap v99: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:21 smithi131 ceph-mon[33041]: pgmap v99: 97 pgs: 97 active+clean; 101 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 224 op/s 2024-04-23T18:20:21.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 191380 11.66 MB/sec execute 16 sec latency 13.838 ms 2024-04-23T18:20:22.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 192948 11.54 MB/sec execute 17 sec latency 14.016 ms 2024-04-23T18:20:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:23 smithi110 ceph-mon[32585]: pgmap v100: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-23T18:20:23.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:23 smithi026 ceph-mon[29364]: pgmap v100: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-23T18:20:23.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:23 smithi131 ceph-mon[33041]: pgmap v100: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-23T18:20:23.664 INFO:tasks.workunit.client.0.smithi026.stdout: 1 195434 11.68 MB/sec execute 18 sec latency 50.627 ms 2024-04-23T18:20:24.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 197496 11.44 MB/sec execute 19 sec latency 13.521 ms 2024-04-23T18:20:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:25 smithi110 ceph-mon[32585]: pgmap v101: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-04-23T18:20:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:25 smithi026 ceph-mon[29364]: pgmap v101: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-04-23T18:20:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:25 smithi131 ceph-mon[33041]: pgmap v101: 97 pgs: 97 active+clean; 104 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-04-23T18:20:25.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 199122 11.47 MB/sec execute 20 sec latency 38.988 ms 2024-04-23T18:20:26.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 201660 11.52 MB/sec execute 21 sec latency 43.017 ms 2024-04-23T18:20:27.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:27 smithi110 ceph-mon[32585]: pgmap v102: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-04-23T18:20:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:27 smithi026 ceph-mon[29364]: pgmap v102: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-04-23T18:20:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:27 smithi131 ceph-mon[33041]: pgmap v102: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.8 MiB/s wr, 408 op/s 2024-04-23T18:20:27.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 203166 11.34 MB/sec execute 22 sec latency 16.743 ms 2024-04-23T18:20:28.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 205978 11.62 MB/sec execute 23 sec latency 10.889 ms 2024-04-23T18:20:29.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:29 smithi110 ceph-mon[32585]: pgmap v103: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:29.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:29 smithi026 ceph-mon[29364]: pgmap v103: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:29.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:29 smithi131 ceph-mon[33041]: pgmap v103: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:29.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 207555 11.38 MB/sec execute 24 sec latency 19.412 ms 2024-04-23T18:20:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 -- 172.21.15.26:0/3907553077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e01031e0 msgr2=0x7f72e01035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 --2- 172.21.15.26:0/3907553077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e01031e0 0x7f72e01035c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009900 tx=0x7f72dc02a510 comp rx=0 tx=0).stop 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 -- 172.21.15.26:0/3907553077 shutdown_connections 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 --2- 172.21.15.26:0/3907553077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e010c520 0x7f72e010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 --2- 172.21.15.26:0/3907553077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e0103b00 0x7f72e010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 --2- 172.21.15.26:0/3907553077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e01031e0 0x7f72e01035c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.353+0000 7f72e7930640 1 -- 172.21.15.26:0/3907553077 >> 172.21.15.26:0/3907553077 conn(0x7f72e0077fd0 msgr2=0x7f72e00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- 172.21.15.26:0/3907553077 shutdown_connections 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- 172.21.15.26:0/3907553077 wait complete. 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 Processor -- start 2024-04-23T18:20:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- start start 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 0x7f72e0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e010c520 0x7f72e019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0112200 con 0x7f72e0103b00 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0112080 con 0x7f72e01031e0 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.354+0000 7f72e7930640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0112380 con 0x7f72e010c520 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 0x7f72e0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56424/0 (socket says 172.21.15.26:56424) 2024-04-23T18:20:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e4ecb640 1 -- 172.21.15.26:0/1446361627 learned_addr learned my addr 172.21.15.26:0/1446361627 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e010c520 0x7f72e019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 -- 172.21.15.26:0/1446361627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e010c520 msgr2=0x7f72e019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e010c520 0x7f72e019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 -- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 msgr2=0x7f72e0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 -- 172.21.15.26:0/1446361627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019d6e0 con 0x7f72e01031e0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e56cc640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 0x7f72e0198740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02ab00 tx=0x7f72dc003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72d67fc640 1 -- 172.21.15.26:0/1446361627 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc03f070 con 0x7f72e01031e0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019d970 con 0x7f72e01031e0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72d67fc640 1 -- 172.21.15.26:0/1446361627 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc031b50 con 0x7f72e01031e0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72d67fc640 1 -- 172.21.15.26:0/1446361627 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc035030 con 0x7f72e01031e0 2024-04-23T18:20:30.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.355+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e01a5260 con 0x7f72e01031e0 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72d67fc640 1 -- 172.21.15.26:0/1446361627 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72dc042070 con 0x7f72e01031e0 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 Processor -- start 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 -- start start 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a06da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72ea647740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x555809fd2970 con 0x55580a06df40 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.358+0000 7f72d67fc640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 0x7f72cc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72ea647740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555809f6f4a0 con 0x55580a06d620 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72ea647740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x555809fce930 con 0x55580a078800 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:39550/0 (socket says 172.21.15.26:39550) 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56438/0 (socket says 172.21.15.26:56438) 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 -- 172.21.15.26:0/2485663858 learned_addr learned my addr 172.21.15.26:0/2485663858 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 0x7f72cc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72d67fc640 1 -- 172.21.15.26:0/1446361627 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f72dc0f0cb0 con 0x7f72e01031e0 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 -- 172.21.15.26:0/2485663858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 msgr2=0x55580a06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a06da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 -- 172.21.15.26:0/2485663858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 msgr2=0x55580a0782c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 -- 172.21.15.26:0/2485663858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555809ce5210 con 0x55580a078800 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e56cc640 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 0x7f72cc0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009fd0 tx=0x7f72d8009de0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72d00115f0 tx=0x7f72d0011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2b3f0679c0b2880 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.359+0000 7f72cb7fe640 1 -- 172.21.15.26:0/2485663858 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d001e070 con 0x55580a078800 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72cb7fe640 1 -- 172.21.15.26:0/2485663858 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d000fb70 con 0x55580a078800 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72cb7fe640 1 -- 172.21.15.26:0/2485663858 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0019ca0 con 0x55580a078800 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 -- 172.21.15.26:0/2485663858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 msgr2=0x55580a07abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72d00115f0 tx=0x7f72d0011ab0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 -- 172.21.15.26:0/2485663858 shutdown_connections 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a07abf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a0782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 --2- 172.21.15.26:0/2485663858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a06da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 -- 172.21.15.26:0/2485663858 >> 172.21.15.26:0/2485663858 conn(0x55580a069b30 msgr2=0x55580a069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 -- 172.21.15.26:0/2485663858 shutdown_connections 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.360+0000 7f72ea647740 1 -- 172.21.15.26:0/2485663858 wait complete. 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 -- start start 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 Processor -- start 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55580a07b430 con 0x55580a06df40 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555809f6d0b0 con 0x55580a06d620 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55580a07b5b0 con 0x55580a078800 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72ea647740 10 client.0 _ll_register_callbacks cb 0x555809f5ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56450/0 (socket says 172.21.15.26:56450) 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:32946/0 (socket says 172.21.15.26:32946) 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e4ecb640 1 -- 172.21.15.26:0/7305884 learned_addr learned my addr 172.21.15.26:0/7305884 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.361+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 -- 172.21.15.26:0/7305884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 msgr2=0x55580a0759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a0759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 -- 172.21.15.26:0/7305884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 msgr2=0x55580a075470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 -- 172.21.15.26:0/7305884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555809db1620 con 0x55580a06d620 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:30.365 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72e56cc640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0ee430 tx=0x7f72dc08b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f89dd774a5ab90 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc049050 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d000fb70 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f72dc0b93a0 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f72dc03b460 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72ea647740 10 client.25499 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55580a087380 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.362+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d000fb70 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f72dc0d5ab0 con 0x55580a06d620 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 10 client.25499 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 10 client.25499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 20 client.25499 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a088880 0x55580a08acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 4 client.25499 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555809fe0390 con 0x55580a088880 2024-04-23T18:20:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a088880 0x55580a08acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a088880 0x55580a08acd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d99da4ec930ee72 server_cookie=d34f89bc487caf3b in_seq=0 out_seq=0 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.363+0000 7f72c8ff9640 10 client.25499 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.364+0000 7f72c8ff9640 1 -- 172.21.15.26:0/7305884 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555809fe0390 con 0x55580a088880 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.364+0000 7f72c8ff9640 10 client.25499 handle_command_reply: tid=0 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.364+0000 7f72ea647740 1 client.25499 shutdown 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 msgr2=0x55580a078180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0ee430 tx=0x7f72dc08b6e0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 shutdown_connections 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a088880 0x55580a08acd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a078800 0x55580a0759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06df40 0x55580a075470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 --2- 172.21.15.26:0/7305884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a078180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72c8ff9640 0 client.25499 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 >> 172.21.15.26:0/7305884 conn(0x55580a069b30 msgr2=0x55580a06f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 shutdown_connections 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 1 -- 172.21.15.26:0/7305884 wait complete. 2024-04-23T18:20:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.365+0000 7f72ea647740 20 client.25499 trim_cache size 0 max 16384 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 Processor -- start 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 -- start start 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a0842f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a077cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01b02c0 con 0x55580a078800 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01adb30 con 0x55580a06d620 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72ea647740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ab980 con 0x55580a06df40 2024-04-23T18:20:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72e5ecd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a0842f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72e5ecd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56462/0 (socket says 172.21.15.26:56462) 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72e5ecd640 1 -- 172.21.15.26:0/2791959828 learned_addr learned my addr 172.21.15.26:0/2791959828 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.380+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a077cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 -- 172.21.15.26:0/2791959828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 msgr2=0x55580a077cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a077cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 -- 172.21.15.26:0/2791959828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 msgr2=0x55580a0905e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 -- 172.21.15.26:0/2791959828 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55580a06e8d0 con 0x55580a06d620 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a077cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:30.379 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72e56cc640 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a0842f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0bca30 tx=0x7f72dc003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc122014a3abd9c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72caffd640 1 -- 172.21.15.26:0/2791959828 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc049040 con 0x55580a06d620 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72caffd640 1 -- 172.21.15.26:0/2791959828 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc0bd210 con 0x55580a06d620 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.381+0000 7f72caffd640 1 -- 172.21.15.26:0/2791959828 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0c5460 con 0x55580a06d620 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- 172.21.15.26:0/2791959828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 msgr2=0x55580a0842f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a0842f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0bca30 tx=0x7f72dc003f90 comp rx=0 tx=0).stop 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- 172.21.15.26:0/2791959828 shutdown_connections 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a078800 0x55580a0905e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a077cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 --2- 172.21.15.26:0/2791959828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a06d620 0x55580a0842f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- 172.21.15.26:0/2791959828 >> 172.21.15.26:0/2791959828 conn(0x55580a069b30 msgr2=0x55580a070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- 172.21.15.26:0/2791959828 shutdown_connections 2024-04-23T18:20:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- 172.21.15.26:0/2791959828 wait complete. 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 -- start start 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.382+0000 7f72ea647740 1 Processor -- start 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 0x55580a080d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a078800 0x55580a07c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01abff0 con 0x55580a06d620 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ad460 con 0x55580a078800 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ac6c0 con 0x55580a06df40 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72ea647740 10 client.0 _ll_register_callbacks cb 0x7f72dc0b9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e56cc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 0x55580a080d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:39574/0 (socket says 172.21.15.26:39574) 2024-04-23T18:20:30.381 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 -- 172.21.15.26:0/219671038 learned_addr learned my addr 172.21.15.26:0/219671038 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e5ecd640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a078800 0x55580a07c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 -- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a078800 msgr2=0x55580a07c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a078800 0x55580a07c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 -- 172.21.15.26:0/219671038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 msgr2=0x55580a080d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 0x55580a080d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 -- 172.21.15.26:0/219671038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55580a0ede50 con 0x55580a06df40 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e56cc640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 0x55580a080d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72e4ecb640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72d0019610 tx=0x7f72d00114f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3280343b33ca0127 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.383+0000 7f72c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d002d020 con 0x55580a06df40 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f72d0005250 con 0x55580a06df40 2024-04-23T18:20:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72dc0bd210 con 0x55580a06df40 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f72d0038050 con 0x55580a06df40 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 10 client.25495 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55580a0ef020 con 0x55580a06df40 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc0bd210 con 0x55580a06df40 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f72d000f7b0 con 0x55580a06df40 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 10 client.25495 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 10 client.25495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 20 client.25495 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a0ef690 0x55580a0f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 4 client.25495 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555809fda200 con 0x55580a0ef690 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.384+0000 7f72e56cc640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a0ef690 0x55580a0f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.385+0000 7f72e56cc640 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a0ef690 0x55580a0f1ab0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb501c64a3b3b81e server_cookie=77f72764269b93da in_seq=0 out_seq=0 2024-04-23T18:20:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.385+0000 7f72c8ff9640 10 client.25495 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.435+0000 7f72c8ff9640 1 -- 172.21.15.26:0/219671038 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555809fda200 con 0x55580a0ef690 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.435+0000 7f72c8ff9640 10 client.25495 handle_command_reply: tid=0 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.435+0000 7f72ea647740 1 client.25495 shutdown 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 msgr2=0x55580a07c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72d0019610 tx=0x7f72d00114f0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 shutdown_connections 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55580a0ef690 0x55580a0f1ab0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55580a078800 0x55580a07c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72c8ff9640 0 client.25495 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55580a06df40 0x55580a07c140 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 --2- 172.21.15.26:0/219671038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55580a06d620 0x55580a080d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 >> 172.21.15.26:0/219671038 conn(0x55580a069b30 msgr2=0x55580a0832e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 shutdown_connections 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 1 -- 172.21.15.26:0/219671038 wait complete. 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72ea647740 20 client.25495 trim_cache size 0 max 16384 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:20:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:20:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 msgr2=0x7f72cc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 0x7f72cc0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009fd0 tx=0x7f72d8009de0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 msgr2=0x7f72e0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 0x7f72e0198740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02ab00 tx=0x7f72dc003b90 comp rx=0 tx=0).stop 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 shutdown_connections 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72cc0a3dc0 0x7f72cc0a6280 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72e010c520 0x7f72e019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72e0103b00 0x7f72e0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 --2- 172.21.15.26:0/1446361627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72e01031e0 0x7f72e0198740 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.436+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 >> 172.21.15.26:0/1446361627 conn(0x7f72e0077fd0 msgr2=0x7f72e00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.437+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 shutdown_connections 2024-04-23T18:20:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:30.437+0000 7f72e7930640 1 -- 172.21.15.26:0/1446361627 wait complete. 2024-04-23T18:20:30.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 209709 11.47 MB/sec execute 25 sec latency 59.301 ms 2024-04-23T18:20:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:31 smithi110 ceph-mon[32585]: pgmap v104: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:31 smithi026 ceph-mon[29364]: pgmap v104: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:31 smithi131 ceph-mon[33041]: pgmap v104: 97 pgs: 97 active+clean; 109 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:20:31.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 211868 11.34 MB/sec execute 26 sec latency 24.611 ms 2024-04-23T18:20:32.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 213504 11.34 MB/sec execute 27 sec latency 18.417 ms 2024-04-23T18:20:32.742 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.8 2024-04-23T18:20:32.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:32.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 -- 172.21.15.26:0/433855978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f601010a0 msgr2=0x7f7f60101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:32.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 --2- 172.21.15.26:0/433855978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f601010a0 0x7f7f60101480 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009900 tx=0x7f7f5002a410 comp rx=0 tx=0).stop 2024-04-23T18:20:32.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 -- 172.21.15.26:0/433855978 shutdown_connections 2024-04-23T18:20:32.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 --2- 172.21.15.26:0/433855978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f60106170 0x7f7f6010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 --2- 172.21.15.26:0/433855978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f60101a50 0x7f7f60105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 --2- 172.21.15.26:0/433855978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f601010a0 0x7f7f60101480 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 -- 172.21.15.26:0/433855978 >> 172.21.15.26:0/433855978 conn(0x7f7f60077f50 msgr2=0x7f7f60078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 -- 172.21.15.26:0/433855978 shutdown_connections 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.858+0000 7f7f6566b640 1 -- 172.21.15.26:0/433855978 wait complete. 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.859+0000 7f7f6566b640 1 Processor -- start 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.859+0000 7f7f6566b640 1 -- start start 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f60106170 0x7f7f60103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f601035b0 0x7f7f601a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7f601110b0 con 0x7f7f60106170 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60110f30 con 0x7f7f601035b0 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f6566b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60111230 con 0x7f7f601010a0 2024-04-23T18:20:32.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:39596/0 (socket says 172.21.15.26:39596) 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 -- 172.21.15.26:0/2952947956 learned_addr learned my addr 172.21.15.26:0/2952947956 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f60106170 0x7f7f60103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5f7fe640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f601035b0 0x7f7f601a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 -- 172.21.15.26:0/2952947956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f601035b0 msgr2=0x7f7f601a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f601035b0 0x7f7f601a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 -- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f60106170 msgr2=0x7f7f60103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f60106170 0x7f7f60103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.860+0000 7f7f5effd640 1 -- 172.21.15.26:0/2952947956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f601a7050 con 0x7f7f601010a0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f5effd640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005940 tx=0x7f7f50004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5003f030 con 0x7f7f601010a0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f50003aa0 con 0x7f7f601010a0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f601a72e0 con 0x7f7f601010a0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50035070 con 0x7f7f601010a0 2024-04-23T18:20:32.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.861+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f601a77d0 con 0x7f7f601010a0 2024-04-23T18:20:32.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.862+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f60068d70 con 0x7f7f601010a0 2024-04-23T18:20:32.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.863+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7f50042030 con 0x7f7f601010a0 2024-04-23T18:20:32.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.864+0000 7f7f5cff9640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 0x7f7f4c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:32.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.864+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f500f0dc0 con 0x7f7f601010a0 2024-04-23T18:20:32.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.864+0000 7f7f567fc640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 0x7f7f4c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:32.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.865+0000 7f7f567fc640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 0x7f7f4c0a6210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f601045c0 tx=0x7f7f4800b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:32.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:32.871+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f500bce40 con 0x7f7f601010a0 2024-04-23T18:20:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.124+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f6010a3b0 con 0x7f7f601010a0 2024-04-23T18:20:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.125+0000 7f7f5cff9640 1 -- 172.21.15.26:0/2952947956 <== mon.1 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11887 (secure 0 0 0) 0x7f7f500c1140 con 0x7f7f601010a0 2024-04-23T18:20:33.119 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:20:33.119 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:19:35.669505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:20:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 msgr2=0x7f7f4c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 0x7f7f4c0a6210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f601045c0 tx=0x7f7f4800b390 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 msgr2=0x7f7f60102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005940 tx=0x7f7f50004270 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 shutdown_connections 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7f4c0a3d50 0x7f7f4c0a6210 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f601035b0 0x7f7f601a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f60106170 0x7f7f60103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 --2- 172.21.15.26:0/2952947956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f601010a0 0x7f7f60102b30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 >> 172.21.15.26:0/2952947956 conn(0x7f7f60077f50 msgr2=0x7f7f600694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 shutdown_connections 2024-04-23T18:20:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.127+0000 7f7f6566b640 1 -- 172.21.15.26:0/2952947956 wait complete. 2024-04-23T18:20:33.132 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 85.8 --await-for 42.0' 2024-04-23T18:20:33.132 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 85.8 --await-for 42.0 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.258+0000 7f76fa9e2640 1 -- 172.21.15.26:0/1412493079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4101240 msgr2=0x7f76f4111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.258+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/1412493079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4101240 0x7f76f4111030 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009cd0 tx=0x7f76ec02a8e0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 -- 172.21.15.26:0/1412493079 shutdown_connections 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/1412493079 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4101240 0x7f76f4111030 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/1412493079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f4100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/1412493079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4102d40 0x7f76f4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 -- 172.21.15.26:0/1412493079 >> 172.21.15.26:0/1412493079 conn(0x7f76f40761d0 msgr2=0x7f76f40765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 -- 172.21.15.26:0/1412493079 shutdown_connections 2024-04-23T18:20:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 -- 172.21.15.26:0/1412493079 wait complete. 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 Processor -- start 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.259+0000 7f76fa9e2640 1 -- start start 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f41a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 0x7f76f419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f76f4114470 con 0x7f76f4102d40 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41142f0 con 0x7f76f4101240 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76fa9e2640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41145f0 con 0x7f76f4100880 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f8f7f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 0x7f76f419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f3fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f41a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:32988/0 (socket says 172.21.15.26:32988) 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 -- 172.21.15.26:0/3221217527 learned_addr learned my addr 172.21.15.26:0/3221217527 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 -- 172.21.15.26:0/3221217527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 msgr2=0x7f76f41a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f41a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 -- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 msgr2=0x7f76f419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 0x7f76f419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 -- 172.21.15.26:0/3221217527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f419afe0 con 0x7f76f4101240 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 0x7f76f419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f3fff640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f41a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.260+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.261+0000 7f76f17fa640 1 -- 172.21.15.26:0/3221217527 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e800cd80 con 0x7f76f4101240 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.261+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f419b2d0 con 0x7f76f4101240 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.261+0000 7f76f17fa640 1 -- 172.21.15.26:0/3221217527 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e80108e0 con 0x7f76f4101240 2024-04-23T18:20:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.261+0000 7f76f17fa640 1 -- 172.21.15.26:0/3221217527 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8018980 con 0x7f76f4101240 2024-04-23T18:20:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.261+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f41a74f0 con 0x7f76f4101240 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 Processor -- start 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f17fa640 1 -- 172.21.15.26:0/3221217527 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f76e8010430 con 0x7f76f4101240 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 -- start start 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a25915a370 con 0x55a259200090 2024-04-23T18:20:33.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a2590f72e0 con 0x55a2591f59c0 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a259156330 con 0x55a2591f50a0 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56518/0 (socket says 172.21.15.26:56518) 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 -- 172.21.15.26:0/4024147857 learned_addr learned my addr 172.21.15.26:0/4024147857 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f3fff640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:33002/0 (socket says 172.21.15.26:33002) 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f17fa640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 0x7f76dc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 -- 172.21.15.26:0/4024147857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 msgr2=0x55a2591f5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 -- 172.21.15.26:0/4024147857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 msgr2=0x55a2591ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.264+0000 7f76f37fe640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76f37fe640 1 -- 172.21.15.26:0/4024147857 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a258f38d80 con 0x55a259200090 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76f3fff640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:33.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 0x7f76dc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76f37fe640 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f76e80ce1a0 tx=0x7f76e80b1160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbc409f817df70d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76f17fa640 1 -- 172.21.15.26:0/3221217527 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76e80d0ab0 con 0x7f76f4101240 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76da7fc640 1 -- 172.21.15.26:0/4024147857 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e809c670 con 0x55a259200090 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76da7fc640 1 -- 172.21.15.26:0/4024147857 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e80d6070 con 0x55a259200090 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.265+0000 7f76da7fc640 1 -- 172.21.15.26:0/4024147857 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e809c490 con 0x55a259200090 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 0x7f76dc0a61e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76ec004af0 tx=0x7f76ec0095c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- 172.21.15.26:0/4024147857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 msgr2=0x55a259202480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f76e80ce1a0 tx=0x7f76e80b1160 comp rx=0 tx=0).stop 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- 172.21.15.26:0/4024147857 shutdown_connections 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a2591ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259202480 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/4024147857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- 172.21.15.26:0/4024147857 >> 172.21.15.26:0/4024147857 conn(0x55a2591f13c0 msgr2=0x55a2591f17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- 172.21.15.26:0/4024147857 shutdown_connections 2024-04-23T18:20:33.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- 172.21.15.26:0/4024147857 wait complete. 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.266+0000 7f76fd7d3740 1 -- start start 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 Processor -- start 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591fd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a259200090 0x55a2591fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a259202e80 con 0x55a2591f59c0 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a2590f4fb0 con 0x55a259200090 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a259203000 con 0x55a2591f50a0 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76fd7d3740 10 client.0 _ll_register_callbacks cb 0x55a2590f49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56532/0 (socket says 172.21.15.26:56532) 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 -- 172.21.15.26:0/3682628799 learned_addr learned my addr 172.21.15.26:0/3682628799 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f3fff640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a259200090 0x55a2591fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 -- 172.21.15.26:0/3682628799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 msgr2=0x55a2591fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591fd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 -- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a259200090 msgr2=0x55a2591fdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a259200090 0x55a2591fdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 -- 172.21.15.26:0/3682628799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a258f99510 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76f37fe640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f76e80ce2e0 tx=0x7f76e801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f59c6d660b9b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.267+0000 7f76d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e809c490 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76e8010a90 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e80d6070 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f76e809c670 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 10 client.16047 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a258e2c670 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e80d6070 con 0x55a2591f59c0 2024-04-23T18:20:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f76e80991a0 con 0x55a2591f59c0 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 10 client.16047 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 10 client.16047 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 20 client.16047 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a2592102d0 0x55a2592126c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 4 client.16047 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a259167d90 con 0x55a2592102d0 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.268+0000 7f76f3fff640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a2592102d0 0x55a2592126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.269+0000 7f76f3fff640 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a2592102d0 0x55a2592126c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7248f2a65d78ef5 server_cookie=a4401d31c97b7a7a in_seq=0 out_seq=0 2024-04-23T18:20:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.269+0000 7f76d37fe640 10 client.16047 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.269+0000 7f76d37fe640 1 -- 172.21.15.26:0/3682628799 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a259167d90 con 0x55a2592102d0 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.269+0000 7f76d37fe640 10 client.16047 handle_command_reply: tid=0 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 client.16047 shutdown 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 msgr2=0x55a2591fd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f76e80ce2e0 tx=0x7f76e801f000 comp rx=0 tx=0).stop 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 shutdown_connections 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a2592102d0 0x55a2592126c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a259200090 0x55a2591fdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a2591f59c0 0x55a2591fd590 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76d37fe640 0 client.16047 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/3682628799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 >> 172.21.15.26:0/3682628799 conn(0x55a2591f13c0 msgr2=0x55a2591f71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 shutdown_connections 2024-04-23T18:20:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 1 -- 172.21.15.26:0/3682628799 wait complete. 2024-04-23T18:20:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.270+0000 7f76fd7d3740 20 client.16047 trim_cache size 0 max 16384 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 Processor -- start 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 -- start start 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a25920b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41b2750 con 0x55a259200090 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41affc0 con 0x55a2591f59c0 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.288+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41ade10 con 0x55a2591f50a0 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f8f7f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f8f7f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:56546/0 (socket says 172.21.15.26:56546) 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a25920b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f8f7f640 1 -- 172.21.15.26:0/2996999639 learned_addr learned my addr 172.21.15.26:0/2996999639 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f3fff640 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 -- 172.21.15.26:0/2996999639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 msgr2=0x55a2591f7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 -- 172.21.15.26:0/2996999639 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 msgr2=0x55a25920bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 -- 172.21.15.26:0/2996999639 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2591fc000 con 0x55a2591f59c0 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f8f7f640 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76f37fe640 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a25920b960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76e809cac0 tx=0x7f76e809b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d988c193dc9a2068 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76d9ffb640 1 -- 172.21.15.26:0/2996999639 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8010720 con 0x55a2591f59c0 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76d9ffb640 1 -- 172.21.15.26:0/2996999639 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e80d6040 con 0x55a2591f59c0 2024-04-23T18:20:33.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.289+0000 7f76d9ffb640 1 -- 172.21.15.26:0/2996999639 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e80d43d0 con 0x55a2591f59c0 2024-04-23T18:20:33.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- 172.21.15.26:0/2996999639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 msgr2=0x55a25920b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a25920b960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76e809cac0 tx=0x7f76e809b040 comp rx=0 tx=0).stop 2024-04-23T18:20:33.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- 172.21.15.26:0/2996999639 shutdown_connections 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f59c0 0x55a25920b960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a25920bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/2996999639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f50a0 0x55a2591f7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- 172.21.15.26:0/2996999639 >> 172.21.15.26:0/2996999639 conn(0x55a2591f13c0 msgr2=0x55a25920b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- 172.21.15.26:0/2996999639 shutdown_connections 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- 172.21.15.26:0/2996999639 wait complete. 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.290+0000 7f76fd7d3740 1 -- start start 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 Processor -- start 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 0x55a259213950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259213ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41ae480 con 0x55a259200090 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41af8f0 con 0x55a2591f50a0 2024-04-23T18:20:33.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41aeb50 con 0x55a2591f59c0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76fd7d3740 10 client.0 _ll_register_callbacks cb 0x7f76e80d4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 0x55a259213950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f8f7f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259213ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:33030/0 (socket says 172.21.15.26:33030) 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f3fff640 1 -- 172.21.15.26:0/1145327610 learned_addr learned my addr 172.21.15.26:0/1145327610 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 -- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 msgr2=0x55a259213410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 -- 172.21.15.26:0/1145327610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 msgr2=0x55a259213ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259213ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.291+0000 7f76f37fe640 1 -- 172.21.15.26:0/1145327610 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2592757a0 con 0x55a2591f59c0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76f3fff640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76f37fe640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 0x55a259213950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55a2591a0230 tx=0x7f76e8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ceb1a16aa51bc27 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e80bb4c0 con 0x55a2591f59c0 2024-04-23T18:20:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76e8098880 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e80d6040 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f76e8010720 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 10 client.25519 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a2592757a0 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e80d6040 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f76e80a5260 con 0x55a2591f59c0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 10 client.25519 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 10 client.25519 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 20 client.25519 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a259276da0 0x55a259279190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 4 client.25519 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.8, "await_for": 42.0}] 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.292+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.8, "await_for": 42.0}) -- 0x55a259161c00 con 0x55a259276da0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.293+0000 7f76f3fff640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a259276da0 0x55a259279190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:33.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.293+0000 7f76f3fff640 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a259276da0 0x55a259279190 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b6d4da37c81b2b server_cookie=222f6f34a79ea90a in_seq=0 out_seq=0 2024-04-23T18:20:33.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.293+0000 7f76d37fe640 10 client.25519 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:33.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.370+0000 7f76d37fe640 1 -- 172.21.15.26:0/1145327610 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a259161c00 con 0x55a259276da0 2024-04-23T18:20:33.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.370+0000 7f76d37fe640 10 client.25519 handle_command_reply: tid=0 2024-04-23T18:20:33.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.370+0000 7f76fd7d3740 1 client.25519 shutdown 2024-04-23T18:20:33.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 msgr2=0x55a259213950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 0x55a259213950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55a2591a0230 tx=0x7f76e8000bc0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 shutdown_connections 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a259276da0 0x55a259279190 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a259200090 0x55a259213ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2591f59c0 0x55a259213950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 --2- 172.21.15.26:0/1145327610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2591f50a0 0x55a259213410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76d37fe640 0 client.25519 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 >> 172.21.15.26:0/1145327610 conn(0x55a2591f13c0 msgr2=0x55a25920bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 shutdown_connections 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 1 -- 172.21.15.26:0/1145327610 wait complete. 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fd7d3740 20 client.25519 trim_cache size 0 max 16384 2024-04-23T18:20:33.365 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 30, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 18, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "f715c2af": { 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 18, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 85.8, 2024-04-23T18:20:33.366 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:33.367 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 msgr2=0x7f76dc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 0x7f76dc0a61e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76ec004af0 tx=0x7f76ec0095c0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 msgr2=0x7f76f41a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.371+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 shutdown_connections 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f76dc0a3d20 0x7f76dc0a61e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76f4101240 0x7f76f41a0be0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f76f4102d40 0x7f76f419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 --2- 172.21.15.26:0/3221217527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76f4100880 0x7f76f41a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 >> 172.21.15.26:0/3221217527 conn(0x7f76f40761d0 msgr2=0x7f76f40fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 shutdown_connections 2024-04-23T18:20:33.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:33.372+0000 7f76fa9e2640 1 -- 172.21.15.26:0/3221217527 wait complete. 2024-04-23T18:20:33.375 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f715c2af', quiesce took 0.6 seconds. Will release after: 25.8 2024-04-23T18:20:33.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:33 smithi110 ceph-mon[32585]: pgmap v105: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 458 op/s 2024-04-23T18:20:33.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:33 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2952947956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:33 smithi026 ceph-mon[29364]: pgmap v105: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 458 op/s 2024-04-23T18:20:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:33 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2952947956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:33 smithi131 ceph-mon[33041]: pgmap v105: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 458 op/s 2024-04-23T18:20:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:33 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2952947956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:33.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 11.18 MB/sec execute 28 sec latency 330.764 ms 2024-04-23T18:20:34.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 10.79 MB/sec execute 29 sec latency 1330.872 ms 2024-04-23T18:20:35.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:35 smithi110 ceph-mon[32585]: pgmap v106: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2024-04-23T18:20:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:35 smithi026 ceph-mon[29364]: pgmap v106: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2024-04-23T18:20:35.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:35 smithi131 ceph-mon[33041]: pgmap v106: 97 pgs: 97 active+clean; 122 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2024-04-23T18:20:35.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 10.43 MB/sec execute 30 sec latency 2330.924 ms 2024-04-23T18:20:36.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 10.10 MB/sec execute 31 sec latency 3331.045 ms 2024-04-23T18:20:37.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:37 smithi110 ceph-mon[32585]: pgmap v107: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-23T18:20:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:37 smithi026 ceph-mon[29364]: pgmap v107: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-23T18:20:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:37 smithi131 ceph-mon[33041]: pgmap v107: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 391 op/s 2024-04-23T18:20:37.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 9.78 MB/sec execute 32 sec latency 4331.114 ms 2024-04-23T18:20:38.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 9.49 MB/sec execute 33 sec latency 5331.232 ms 2024-04-23T18:20:39.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:39 smithi110 ceph-mon[32585]: pgmap v108: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:39 smithi026 ceph-mon[29364]: pgmap v108: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:39.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:39 smithi131 ceph-mon[33041]: pgmap v108: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:39.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 9.21 MB/sec execute 34 sec latency 6331.339 ms 2024-04-23T18:20:40.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 8.94 MB/sec execute 35 sec latency 7331.393 ms 2024-04-23T18:20:41.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:41 smithi110 ceph-mon[32585]: pgmap v109: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:41.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:41 smithi026 ceph-mon[29364]: pgmap v109: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:41.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:41 smithi131 ceph-mon[33041]: pgmap v109: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 258 op/s 2024-04-23T18:20:41.665 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 8.70 MB/sec execute 36 sec latency 8331.500 ms 2024-04-23T18:20:42.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 8.46 MB/sec execute 37 sec latency 9331.546 ms 2024-04-23T18:20:43.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:43 smithi110 ceph-mon[32585]: pgmap v110: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 289 op/s 2024-04-23T18:20:43.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:43 smithi026 ceph-mon[29364]: pgmap v110: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 289 op/s 2024-04-23T18:20:43.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:43 smithi131 ceph-mon[33041]: pgmap v110: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 289 op/s 2024-04-23T18:20:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 -- 172.21.15.26:0/3803649329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c101aa0 msgr2=0x7f744c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 --2- 172.21.15.26:0/3803649329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c101aa0 0x7f744c105a90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7440009900 tx=0x7f744002a410 comp rx=0 tx=0).stop 2024-04-23T18:20:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 -- 172.21.15.26:0/3803649329 shutdown_connections 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 --2- 172.21.15.26:0/3803649329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 0x7f744c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 --2- 172.21.15.26:0/3803649329 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c101aa0 0x7f744c105a90 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 --2- 172.21.15.26:0/3803649329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 0x7f744c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 -- 172.21.15.26:0/3803649329 >> 172.21.15.26:0/3803649329 conn(0x7f744c077fd0 msgr2=0x7f744c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 -- 172.21.15.26:0/3803649329 shutdown_connections 2024-04-23T18:20:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.536+0000 7f7453d84640 1 -- 172.21.15.26:0/3803649329 wait complete. 2024-04-23T18:20:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 Processor -- start 2024-04-23T18:20:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 -- start start 2024-04-23T18:20:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 0x7f744c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 0x7f744c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1121b0 con 0x7f744c19cec0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f744c112030 con 0x7f744c106170 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7453d84640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f744c112330 con 0x7f744c1010f0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7452321640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7452321640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55888/0 (socket says 172.21.15.26:55888) 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7452321640 1 -- 172.21.15.26:0/3053479320 learned_addr learned my addr 172.21.15.26:0/3053479320 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f745131f640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 0x7f744c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7451b20640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 0x7f744c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7452321640 1 -- 172.21.15.26:0/3053479320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 msgr2=0x7f744c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.537+0000 7f7452321640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 0x7f744c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7452321640 1 -- 172.21.15.26:0/3053479320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 msgr2=0x7f744c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7452321640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 0x7f744c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7452321640 1 -- 172.21.15.26:0/3053479320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c19d9b0 con 0x7f744c19cec0 2024-04-23T18:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7452321640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f743c00d750 tx=0x7f743c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f744affd640 1 -- 172.21.15.26:0/3053479320 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c017070 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f744affd640 1 -- 172.21.15.26:0/3053479320 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743c0054d0 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744c19dca0 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f744affd640 1 -- 172.21.15.26:0/3053479320 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c004c60 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.538+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744c1a5460 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.541+0000 7f744affd640 1 -- 172.21.15.26:0/3053479320 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f743c009bd0 con 0x7f744c19cec0 2024-04-23T18:20:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 Processor -- start 2024-04-23T18:20:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 -- start start 2024-04-23T18:20:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc6596900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ddc64fb970 con 0x55ddc65a5c10 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ddc64984a0 con 0x55ddc6596520 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7456b78740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ddc64f7930 con 0x55ddc6596e40 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f745131f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f744affd640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 0x7f74340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f745131f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55898/0 (socket says 172.21.15.26:55898) 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f745131f640 1 -- 172.21.15.26:0/1535273881 learned_addr learned my addr 172.21.15.26:0/1535273881 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7451b20640 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f745131f640 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc6596900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.542+0000 7f7452321640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 0x7f74340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f744affd640 1 -- 172.21.15.26:0/3053479320 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f743c0d00b0 con 0x7f744c19cec0 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 -- 172.21.15.26:0/1535273881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 msgr2=0x55ddc65a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 -- 172.21.15.26:0/1535273881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 msgr2=0x55ddc6596900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc6596900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 -- 172.21.15.26:0/1535273881 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc620e210 con 0x55ddc65a5c10 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f745131f640 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7440009900 tx=0x7f7440005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad156ac9fe061d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f7452321640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 0x7f74340a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f743c0984e0 tx=0x7f743c009f00 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f7433fff640 1 -- 172.21.15.26:0/1535273881 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440035070 con 0x55ddc65a5c10 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f7433fff640 1 -- 172.21.15.26:0/1535273881 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74400035f0 con 0x55ddc65a5c10 2024-04-23T18:20:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.543+0000 7f7433fff640 1 -- 172.21.15.26:0/1535273881 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74400044e0 con 0x55ddc65a5c10 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- 172.21.15.26:0/1535273881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 msgr2=0x55ddc65a8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7440009900 tx=0x7f7440005780 comp rx=0 tx=0).stop 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- 172.21.15.26:0/1535273881 shutdown_connections 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc65a5c10 0x55ddc65a8000 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 --2- 172.21.15.26:0/1535273881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc6596900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- 172.21.15.26:0/1535273881 >> 172.21.15.26:0/1535273881 conn(0x55ddc6592b30 msgr2=0x55ddc6592f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- 172.21.15.26:0/1535273881 shutdown_connections 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- 172.21.15.26:0/1535273881 wait complete. 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.544+0000 7f7456b78740 1 -- start start 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 Processor -- start 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ddc65a87e0 con 0x55ddc6596520 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ddc64960b0 con 0x55ddc65a5c10 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ddc65a8960 con 0x55ddc6596e40 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7456b78740 10 client.0 _ll_register_callbacks cb 0x55ddc6483d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55902/0 (socket says 172.21.15.26:55902) 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 -- 172.21.15.26:0/117737977 learned_addr learned my addr 172.21.15.26:0/117737977 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f7451b20640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 -- 172.21.15.26:0/117737977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 msgr2=0x55ddc65a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.545+0000 7f745131f640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f745131f640 1 -- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 msgr2=0x55ddc65a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f745131f640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f745131f640 1 -- 172.21.15.26:0/117737977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc62da620 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7451b20640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f745131f640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7440000c00 tx=0x7f7440003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c405f2c3b7ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440041070 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f74400039b0 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74400035f0 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7440035040 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7456b78740 10 client.16065 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ddc65b47f0 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74400035f0 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f74400046c0 con 0x55ddc6596520 2024-04-23T18:20:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7456b78740 10 client.16065 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.546+0000 7f7456b78740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7456b78740 20 client.16065 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc65b5b40 0x55ddc65b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7456b78740 4 client.16065 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddc6509390 con 0x55ddc65b5b40 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7451b20640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc65b5b40 0x55ddc65b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f7451b20640 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc65b5b40 0x55ddc65b7f30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9271f520a58ced0 server_cookie=20057bc40fed40c in_seq=0 out_seq=0 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.547+0000 7f74317fa640 10 client.16065 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f74317fa640 1 -- 172.21.15.26:0/117737977 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddc6509390 con 0x55ddc65b5b40 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f74317fa640 10 client.16065 handle_command_reply: tid=0 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 client.16065 shutdown 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 msgr2=0x55ddc65a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7440000c00 tx=0x7f7440003f50 comp rx=0 tx=0).stop 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 shutdown_connections 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc65b5b40 0x55ddc65b7f30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596e40 0x55ddc65a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 --2- 172.21.15.26:0/117737977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596520 0x55ddc65a55e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f74317fa640 0 client.16065 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 >> 172.21.15.26:0/117737977 conn(0x55ddc6592b30 msgr2=0x55ddc65989a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 shutdown_connections 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.548+0000 7f7456b78740 1 -- 172.21.15.26:0/117737977 wait complete. 2024-04-23T18:20:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.549+0000 7f7456b78740 20 client.16065 trim_cache size 0 max 16384 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.563+0000 7f7456b78740 1 Processor -- start 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 -- start start 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc65b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc65a5c10 0x55ddc659fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1ae010 con 0x55ddc6596e40 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1ab880 con 0x55ddc6596520 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7456b78740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1a9710 con 0x55ddc65a5c10 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55918/0 (socket says 172.21.15.26:55918) 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 -- 172.21.15.26:0/3150792104 learned_addr learned my addr 172.21.15.26:0/3150792104 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f745131f640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc65a5c10 0x55ddc659fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f745131f640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc65b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 -- 172.21.15.26:0/3150792104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc65a5c10 msgr2=0x55ddc659fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc65a5c10 0x55ddc659fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 -- 172.21.15.26:0/3150792104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 msgr2=0x55ddc65b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc65b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 -- 172.21.15.26:0/3150792104 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc6596c70 con 0x55ddc6596e40 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f745131f640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc65b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.564+0000 7f7451b20640 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55ddc65a89d0 tx=0x7f7444007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8ab4d10ea1431b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f74337fe640 1 -- 172.21.15.26:0/3150792104 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444013c10 con 0x55ddc6596e40 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f74337fe640 1 -- 172.21.15.26:0/3150792104 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f744401e070 con 0x55ddc6596e40 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f74337fe640 1 -- 172.21.15.26:0/3150792104 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744401a870 con 0x55ddc6596e40 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 -- 172.21.15.26:0/3150792104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 msgr2=0x55ddc65a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55ddc65a89d0 tx=0x7f7444007700 comp rx=0 tx=0).stop 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 -- 172.21.15.26:0/3150792104 shutdown_connections 2024-04-23T18:20:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc65a5c10 0x55ddc659fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65a5110 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 --2- 172.21.15.26:0/3150792104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc6596520 0x55ddc65b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 -- 172.21.15.26:0/3150792104 >> 172.21.15.26:0/3150792104 conn(0x55ddc6592b30 msgr2=0x55ddc65b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 -- 172.21.15.26:0/3150792104 shutdown_connections 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.565+0000 7f7456b78740 1 -- 172.21.15.26:0/3150792104 wait complete. 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 -- start start 2024-04-23T18:20:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 Processor -- start 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596520 0x55ddc65bb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1a9d60 con 0x55ddc6596e40 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1ab1b0 con 0x55ddc65a5c10 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f744c1aa410 con 0x55ddc6596520 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7456b78740 10 client.0 _ll_register_callbacks cb 0x7f7440031740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55930/0 (socket says 172.21.15.26:55930) 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 -- 172.21.15.26:0/1968485768 learned_addr learned my addr 172.21.15.26:0/1968485768 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f745131f640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596520 0x55ddc65bb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f745131f640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 -- 172.21.15.26:0/1968485768 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596520 msgr2=0x55ddc65bb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596520 0x55ddc65bb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 -- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 msgr2=0x55ddc65b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.566+0000 7f7451b20640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7451b20640 1 -- 172.21.15.26:0/1968485768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc65b9370 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f745131f640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7451b20640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7444005c10 tx=0x7f744401aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8306ebf4eac0a41e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444003970 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7444003b10 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f744401e070 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7444003d50 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 10 client.16071 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ddc661bc20 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f744401e070 con 0x55ddc6596e40 2024-04-23T18:20:43.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7444023640 con 0x55ddc6596e40 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 10 client.16071 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 20 client.16071 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.567+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc661c360 0x55ddc661e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.568+0000 7f7456b78740 4 client.16071 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.568+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ddc6503200 con 0x55ddc661c360 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.568+0000 7f745131f640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc661c360 0x55ddc661e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.568+0000 7f745131f640 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc661c360 0x55ddc661e780 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461e0dc025ba8805 server_cookie=567fa5c452bad7a0 in_seq=0 out_seq=0 2024-04-23T18:20:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.568+0000 7f74317fa640 10 client.16071 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f74317fa640 1 -- 172.21.15.26:0/1968485768 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ddc6503200 con 0x55ddc661c360 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f74317fa640 10 client.16071 handle_command_reply: tid=0 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 client.16071 shutdown 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 msgr2=0x55ddc65b8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7444005c10 tx=0x7f744401aa30 comp rx=0 tx=0).stop 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 shutdown_connections 2024-04-23T18:20:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ddc661c360 0x55ddc661e780 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ddc65a5c10 0x55ddc65b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ddc6596e40 0x55ddc65b8680 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f74317fa640 0 client.16071 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 --2- 172.21.15.26:0/1968485768 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ddc6596520 0x55ddc65bb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 >> 172.21.15.26:0/1968485768 conn(0x55ddc6592b30 msgr2=0x55ddc6598900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 shutdown_connections 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 1 -- 172.21.15.26:0/1968485768 wait complete. 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.600+0000 7f7456b78740 20 client.16071 trim_cache size 0 max 16384 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:20:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:20:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 msgr2=0x7f74340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 0x7f74340a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f743c0984e0 tx=0x7f743c009f00 comp rx=0 tx=0).stop 2024-04-23T18:20:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 msgr2=0x7f744c19d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f743c00d750 tx=0x7f743c00dc10 comp rx=0 tx=0).stop 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 shutdown_connections 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f74340a3e40 0x7f74340a6300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f744c106170 0x7f744c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f744c19cec0 0x7f744c19d370 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 --2- 172.21.15.26:0/3053479320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f744c1010f0 0x7f744c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 >> 172.21.15.26:0/3053479320 conn(0x7f744c077fd0 msgr2=0x7f744c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 shutdown_connections 2024-04-23T18:20:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:43.601+0000 7f7453d84640 1 -- 172.21.15.26:0/3053479320 wait complete. 2024-04-23T18:20:43.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 8.24 MB/sec execute 38 sec latency 10331.683 ms 2024-04-23T18:20:44.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 8.03 MB/sec execute 39 sec latency 11331.749 ms 2024-04-23T18:20:45.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:45 smithi110 ceph-mon[32585]: pgmap v111: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T18:20:45.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:45 smithi026 ceph-mon[29364]: pgmap v111: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T18:20:45.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:45 smithi131 ceph-mon[33041]: pgmap v111: 97 pgs: 97 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T18:20:45.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 7.83 MB/sec execute 40 sec latency 12331.798 ms 2024-04-23T18:20:46.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 7.63 MB/sec execute 41 sec latency 13331.893 ms 2024-04-23T18:20:47.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:47 smithi110 ceph-mon[32585]: pgmap v112: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 170 op/s 2024-04-23T18:20:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:47 smithi026 ceph-mon[29364]: pgmap v112: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 170 op/s 2024-04-23T18:20:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:47 smithi131 ceph-mon[33041]: pgmap v112: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 170 op/s 2024-04-23T18:20:47.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 7.45 MB/sec execute 42 sec latency 14331.956 ms 2024-04-23T18:20:48.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 7.28 MB/sec execute 43 sec latency 15332.100 ms 2024-04-23T18:20:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:49 smithi110 ceph-mon[32585]: pgmap v113: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:49.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:49 smithi026 ceph-mon[29364]: pgmap v113: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:49.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:49 smithi131 ceph-mon[33041]: pgmap v113: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:49.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 7.11 MB/sec execute 44 sec latency 16332.195 ms 2024-04-23T18:20:50.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.96 MB/sec execute 45 sec latency 17332.258 ms 2024-04-23T18:20:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:51 smithi110 ceph-mon[32585]: pgmap v114: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:51.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:51 smithi026 ceph-mon[29364]: pgmap v114: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:51 smithi131 ceph-mon[33041]: pgmap v114: 97 pgs: 97 active+clean; 118 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-04-23T18:20:51.666 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.80 MB/sec execute 46 sec latency 18332.380 ms 2024-04-23T18:20:52.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.66 MB/sec execute 47 sec latency 19332.483 ms 2024-04-23T18:20:53.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:53 smithi110 ceph-mon[32585]: pgmap v115: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:20:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:53 smithi026 ceph-mon[29364]: pgmap v115: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:20:53.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:53 smithi131 ceph-mon[33041]: pgmap v115: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:20:53.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.52 MB/sec execute 48 sec latency 20332.595 ms 2024-04-23T18:20:54.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.39 MB/sec execute 49 sec latency 21332.690 ms 2024-04-23T18:20:55.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:55 smithi110 ceph-mon[32585]: pgmap v116: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:20:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:55 smithi026 ceph-mon[29364]: pgmap v116: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:20:55.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:55 smithi131 ceph-mon[33041]: pgmap v116: 97 pgs: 97 active+clean; 116 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:20:55.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.26 MB/sec execute 50 sec latency 22332.739 ms 2024-04-23T18:20:56.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.14 MB/sec execute 51 sec latency 23332.775 ms 2024-04-23T18:20:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 -- 172.21.15.26:0/1323198836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f14103b00 msgr2=0x7f7f1410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 --2- 172.21.15.26:0/1323198836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f14103b00 0x7f7f1410bfe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009900 tx=0x7f7f0802a480 comp rx=0 tx=0).stop 2024-04-23T18:20:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 -- 172.21.15.26:0/1323198836 shutdown_connections 2024-04-23T18:20:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 --2- 172.21.15.26:0/1323198836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f1410c520 0x7f7f1410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 --2- 172.21.15.26:0/1323198836 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f14103b00 0x7f7f1410bfe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 --2- 172.21.15.26:0/1323198836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141031e0 0x7f7f141035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 -- 172.21.15.26:0/1323198836 >> 172.21.15.26:0/1323198836 conn(0x7f7f14078000 msgr2=0x7f7f14078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 -- 172.21.15.26:0/1323198836 shutdown_connections 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 -- 172.21.15.26:0/1323198836 wait complete. 2024-04-23T18:20:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.700+0000 7f7f19e42640 1 Processor -- start 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 -- start start 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f141031e0 0x7f7f141a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f14103b00 0x7f7f141a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14112190 con 0x7f7f1410c520 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14112010 con 0x7f7f14103b00 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f19e42640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14112310 con 0x7f7f141031e0 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51890/0 (socket says 172.21.15.26:51890) 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 -- 172.21.15.26:0/2679672434 learned_addr learned my addr 172.21.15.26:0/2679672434 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f14103b00 0x7f7f141a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f141031e0 0x7f7f141a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 -- 172.21.15.26:0/2679672434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f141031e0 msgr2=0x7f7f141a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f141031e0 0x7f7f141a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 -- 172.21.15.26:0/2679672434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f14103b00 msgr2=0x7f7f141a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f14103b00 0x7f7f141a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.701+0000 7f7f13fff640 1 -- 172.21.15.26:0/2679672434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1419af20 con 0x7f7f1410c520 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f13fff640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cce0 tx=0x7f7f0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f10ff9640 1 -- 172.21.15.26:0/2679672434 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00f450 con 0x7f7f1410c520 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f10ff9640 1 -- 172.21.15.26:0/2679672434 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c00faa0 con 0x7f7f1410c520 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1419b210 con 0x7f7f1410c520 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f10ff9640 1 -- 172.21.15.26:0/2679672434 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c018b70 con 0x7f7f1410c520 2024-04-23T18:20:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.702+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f141a7520 con 0x7f7f1410c520 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.704+0000 7f7f10ff9640 1 -- 172.21.15.26:0/2679672434 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7f0c018d10 con 0x7f7f1410c520 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.704+0000 7f7f1cc31740 1 Processor -- start 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.704+0000 7f7f1cc31740 1 -- start start 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71646a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f716512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x555f715ab970 con 0x555f71651800 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f715484a0 con 0x555f71646620 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x555f715a7930 con 0x555f71646f40 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51900/0 (socket says 172.21.15.26:51900) 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f10ff9640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 0x7f7efc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 -- 172.21.15.26:0/3395067647 learned_addr learned my addr 172.21.15.26:0/3395067647 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f137fe640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f716512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71646a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f10ff9640 1 -- 172.21.15.26:0/2679672434 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f0c0d0ab0 con 0x7f7f1410c520 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f13fff640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 0x7f7efc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 -- 172.21.15.26:0/3395067647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 msgr2=0x555f716512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f716512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 -- 172.21.15.26:0/3395067647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 msgr2=0x555f71646a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71646a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 -- 172.21.15.26:0/3395067647 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f712be210 con 0x555f71651800 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f137fe640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f716512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.705+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7f080096c0 tx=0x7f7f08009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e94ca983ad3859 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f13fff640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 0x7f7efc0a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ccb0 tx=0x7f7f0c00af30 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7ef9ffb640 1 -- 172.21.15.26:0/3395067647 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08035070 con 0x555f71651800 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7ef9ffb640 1 -- 172.21.15.26:0/3395067647 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080316f0 con 0x555f71651800 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7ef9ffb640 1 -- 172.21.15.26:0/3395067647 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080047a0 con 0x555f71651800 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 -- 172.21.15.26:0/3395067647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 msgr2=0x555f71653bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7f080096c0 tx=0x7f7f08009610 comp rx=0 tx=0).stop 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 -- 172.21.15.26:0/3395067647 shutdown_connections 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71651800 0x555f71653bf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f716512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/3395067647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71646a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.706+0000 7f7f1cc31740 1 -- 172.21.15.26:0/3395067647 >> 172.21.15.26:0/3395067647 conn(0x555f71642b30 msgr2=0x555f71642f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- 172.21.15.26:0/3395067647 shutdown_connections 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- 172.21.15.26:0/3395067647 wait complete. 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- start start 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 Processor -- start 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71651130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 0x555f7164e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x555f716543d0 con 0x555f71646f40 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f715460b0 con 0x555f71646620 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x555f71654550 con 0x555f71651800 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.707+0000 7f7f1cc31740 10 client.0 _ll_register_callbacks cb 0x555f71533d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51910/0 (socket says 172.21.15.26:51910) 2024-04-23T18:20:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 -- 172.21.15.26:0/1614341641 learned_addr learned my addr 172.21.15.26:0/1614341641 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71651130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 0x555f7164e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 -- 172.21.15.26:0/1614341641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 msgr2=0x555f7164e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 0x555f7164e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 -- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 msgr2=0x555f71651130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71651130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 -- 172.21.15.26:0/1614341641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f7138a620 con 0x555f71646f40 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 0x555f7164e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f137fe640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7f04012810 tx=0x7f7f04012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12414820af02ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7ef2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04010dc0 con 0x555f71646f40 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f04004020 con 0x555f71646f40 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f080316f0 con 0x555f71646f40 2024-04-23T18:20:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7f04004220 con 0x555f71646f40 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f1cc31740 10 client.16083 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.708+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f71660380 con 0x555f71646f40 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080316f0 con 0x555f71646f40 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7f04021070 con 0x555f71646f40 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 10 client.16083 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 20 client.16083 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71661840 0x555f71663c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 4 client.16083 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f715b9390 con 0x555f71661840 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.709+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71661840 0x555f71663c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.710+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71661840 0x555f71663c50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=239d7dd362e9ffd2 server_cookie=87d63ee0661d6546 in_seq=0 out_seq=0 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.710+0000 7f7ef2ffd640 10 client.16083 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.710+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/1614341641 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f715b9390 con 0x555f71661840 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.710+0000 7f7ef2ffd640 10 client.16083 handle_command_reply: tid=0 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.710+0000 7f7f1cc31740 1 client.16083 shutdown 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 msgr2=0x555f7164e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7f04012810 tx=0x7f7f04012cd0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 shutdown_connections 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71661840 0x555f71663c50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71651800 0x555f7164e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7ef2ffd640 0 client.16083 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646f40 0x555f7164e420 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/1614341641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646620 0x555f71651130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 >> 172.21.15.26:0/1614341641 conn(0x555f71642b30 msgr2=0x555f71648860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 shutdown_connections 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 1 -- 172.21.15.26:0/1614341641 wait complete. 2024-04-23T18:20:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.711+0000 7f7f1cc31740 20 client.16083 trim_cache size 0 max 16384 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 Processor -- start 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 -- start start 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71642b30 0x555f7165d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 0x555f71650c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141b24c0 con 0x555f71646620 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141afd30 con 0x555f71646f40 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141adb80 con 0x555f71642b30 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51926/0 (socket says 172.21.15.26:51926) 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.726+0000 7f7f137fe640 1 -- 172.21.15.26:0/2581681545 learned_addr learned my addr 172.21.15.26:0/2581681545 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 0x555f71650c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 -- 172.21.15.26:0/2581681545 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71642b30 msgr2=0x555f7165d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71642b30 0x555f7165d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 -- 172.21.15.26:0/2581681545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 msgr2=0x555f71650c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 0x555f71650c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 -- 172.21.15.26:0/2581681545 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f7164efb0 con 0x555f71646620 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 0x555f71650c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7f040058c0 tx=0x7f7f04010820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c245fdb3f916678 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7ef97fa640 1 -- 172.21.15.26:0/2581681545 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04031070 con 0x555f71646620 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7ef97fa640 1 -- 172.21.15.26:0/2581681545 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f04028040 con 0x555f71646620 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.727+0000 7f7ef97fa640 1 -- 172.21.15.26:0/2581681545 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04027070 con 0x555f71646620 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2581681545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 msgr2=0x555f71649970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7f040058c0 tx=0x7f7f04010820 comp rx=0 tx=0).stop 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2581681545 shutdown_connections 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71646f40 0x555f71650c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f71649970 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2581681545 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71642b30 0x555f7165d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2581681545 >> 172.21.15.26:0/2581681545 conn(0x555f7164eac0 msgr2=0x555f71649260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2581681545 shutdown_connections 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2581681545 wait complete. 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 -- start start 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.728+0000 7f7f1cc31740 1 Processor -- start 2024-04-23T18:20:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71642b30 0x555f71655390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f71655e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ae1f0 con 0x555f71646620 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141af660 con 0x555f71642b30 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ae8c0 con 0x555f71646f40 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 10 client.0 _ll_register_callbacks cb 0x7f7f0400af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:51942/0 (socket says 172.21.15.26:51942) 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 -- 172.21.15.26:0/2369717142 learned_addr learned my addr 172.21.15.26:0/2369717142 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71642b30 0x555f71655390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f71655e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 -- 172.21.15.26:0/2369717142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 msgr2=0x555f71655e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f71655e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 -- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71642b30 msgr2=0x555f71655390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71642b30 0x555f71655390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 -- 172.21.15.26:0/2369717142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f716c7020 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f71655e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f137fe640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7f04004ce0 tx=0x7f7f04005010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5db2a23b80e353 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7ef2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04031040 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f04004920 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f04028040 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7f04027040 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.729+0000 7f7f1cc31740 10 client.16089 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f716c8260 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f04028040 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7f04034070 con 0x555f71646620 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 10 client.16089 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 20 client.16089 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71651800 0x555f716ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 4 client.16089 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f715b3200 con 0x555f71651800 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.730+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71651800 0x555f716ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.731+0000 7f7f12ffd640 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71651800 0x555f716ca870 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7f524e57ef5a7f server_cookie=727c1a28994406e2 in_seq=0 out_seq=0 2024-04-23T18:20:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.731+0000 7f7ef2ffd640 10 client.16089 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7ef2ffd640 1 -- 172.21.15.26:0/2369717142 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f715b3200 con 0x555f71651800 2024-04-23T18:20:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7ef2ffd640 10 client.16089 handle_command_reply: tid=0 2024-04-23T18:20:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7f1cc31740 1 client.16089 shutdown 2024-04-23T18:20:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 msgr2=0x555f716558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7f04004ce0 tx=0x7f7f04005010 comp rx=0 tx=0).stop 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 shutdown_connections 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.751+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x555f71651800 0x555f716ca870 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f71646f40 0x555f71655e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x555f71646620 0x555f716558d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 --2- 172.21.15.26:0/2369717142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f71642b30 0x555f71655390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7ef2ffd640 0 client.16089 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 >> 172.21.15.26:0/2369717142 conn(0x555f7164eac0 msgr2=0x555f71641cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 shutdown_connections 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 1 -- 172.21.15.26:0/2369717142 wait complete. 2024-04-23T18:20:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f1cc31740 20 client.16089 trim_cache size 0 max 16384 2024-04-23T18:20:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:20:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:20:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:20:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 msgr2=0x7f7efc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 0x7f7efc0a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ccb0 tx=0x7f7f0c00af30 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 msgr2=0x7f7f1419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cce0 tx=0x7f7f0c00a620 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 shutdown_connections 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7efc0a3e40 0x7f7efc0a6300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f14103b00 0x7f7f141a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7f1410c520 0x7f7f1419a660 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 --2- 172.21.15.26:0/2679672434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f141031e0 0x7f7f141a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 >> 172.21.15.26:0/2679672434 conn(0x7f7f14078000 msgr2=0x7f7f14108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 shutdown_connections 2024-04-23T18:20:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:20:56.752+0000 7f7f19e42640 1 -- 172.21.15.26:0/2679672434 wait complete. 2024-04-23T18:20:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:57 smithi110 ceph-mon[32585]: pgmap v117: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 94 op/s 2024-04-23T18:20:57.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:57 smithi026 ceph-mon[29364]: pgmap v117: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 94 op/s 2024-04-23T18:20:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:57 smithi131 ceph-mon[33041]: pgmap v117: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 94 op/s 2024-04-23T18:20:57.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 6.02 MB/sec execute 52 sec latency 24332.809 ms 2024-04-23T18:20:58.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:58 smithi131 ceph-mon[33041]: pgmap v118: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:20:58.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 5.91 MB/sec execute 53 sec latency 25332.918 ms 2024-04-23T18:20:58.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:58 smithi110 ceph-mon[32585]: pgmap v118: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:20:58.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:58 smithi026 ceph-mon[29364]: pgmap v118: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:20:59.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 -- 172.21.15.26:0/3523189381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd040102b60 msgr2=0x7fd040106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 --2- 172.21.15.26:0/3523189381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd040102b60 0x7fd040106ba0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd028009900 tx=0x7fd02802a410 comp rx=0 tx=0).stop 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 -- 172.21.15.26:0/3523189381 shutdown_connections 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 --2- 172.21.15.26:0/3523189381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd040107280 0x7fd04010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 --2- 172.21.15.26:0/3523189381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd040102b60 0x7fd040106ba0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 --2- 172.21.15.26:0/3523189381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0401021b0 0x7fd040102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 -- 172.21.15.26:0/3523189381 >> 172.21.15.26:0/3523189381 conn(0x7fd0400fd9f0 msgr2=0x7fd0400ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 -- 172.21.15.26:0/3523189381 shutdown_connections 2024-04-23T18:20:59.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.298+0000 7fd045459640 1 -- 172.21.15.26:0/3523189381 wait complete. 2024-04-23T18:20:59.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 Processor -- start 2024-04-23T18:20:59.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 -- start start 2024-04-23T18:20:59.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd040107280 0x7fd04019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0401985e0 0x7fd040198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd0401121c0 con 0x7fd0401021b0 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd040112040 con 0x7fd040107280 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd045459640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd040112340 con 0x7fd0401985e0 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd03effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd03effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59960/0 (socket says 172.21.15.26:59960) 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd03effd640 1 -- 172.21.15.26:0/681527154 learned_addr learned my addr 172.21.15.26:0/681527154 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.299+0000 7fd03e7fc640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd040107280 0x7fd04019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 -- 172.21.15.26:0/681527154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0401985e0 msgr2=0x7fd040198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0401985e0 0x7fd040198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 -- 172.21.15.26:0/681527154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd040107280 msgr2=0x7fd04019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd040107280 0x7fd04019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 -- 172.21.15.26:0/681527154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd040199100 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd03effd640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd03000eab0 tx=0x7fd03000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000ce40 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd030011920 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030010a30 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0401993f0 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.300+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0401a55a0 con 0x7fd0401021b0 2024-04-23T18:20:59.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.301+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd040068d70 con 0x7fd0401021b0 2024-04-23T18:20:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.302+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd030010bd0 con 0x7fd0401021b0 2024-04-23T18:20:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.303+0000 7fd037fff640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 0x7fd02c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.303+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd0300d0e00 con 0x7fd0401021b0 2024-04-23T18:20:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.303+0000 7fd03e7fc640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 0x7fd02c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.303+0000 7fd03e7fc640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 0x7fd02c0a61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd028000c00 tx=0x7fd028031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.306+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd030016030 con 0x7fd0401021b0 2024-04-23T18:20:59.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.533+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd040199ac0 con 0x7fd0401021b0 2024-04-23T18:20:59.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.534+0000 7fd037fff640 1 -- 172.21.15.26:0/681527154 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11887 (secure 0 0 0) 0x7fd03009d4d0 con 0x7fd0401021b0 2024-04-23T18:20:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:20:59.529 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:19:35.669505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:20:59.529 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 30 2024-04-23T18:20:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 msgr2=0x7fd02c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 0x7fd02c0a61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd028000c00 tx=0x7fd028031400 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 msgr2=0x7fd04019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd03000eab0 tx=0x7fd03000ef70 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 shutdown_connections 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd02c0a3d20 0x7fd02c0a61e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd040107280 0x7fd04019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0401985e0 0x7fd040198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 --2- 172.21.15.26:0/681527154 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd0401021b0 0x7fd04019e3e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 >> 172.21.15.26:0/681527154 conn(0x7fd0400fd9f0 msgr2=0x7fd0400fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 shutdown_connections 2024-04-23T18:20:59.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.536+0000 7fd045459640 1 -- 172.21.15.26:0/681527154 wait complete. 2024-04-23T18:20:59.541 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id f715c2af --release --await' 2024-04-23T18:20:59.541 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id f715c2af --release --await 2024-04-23T18:20:59.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- 172.21.15.26:0/506074267 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100450 msgr2=0x7fc7201008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 --2- 172.21.15.26:0/506074267 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100450 0x7fc7201008d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc714009930 tx=0x7fc71402a420 comp rx=0 tx=0).stop 2024-04-23T18:20:59.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- 172.21.15.26:0/506074267 shutdown_connections 2024-04-23T18:20:59.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 --2- 172.21.15.26:0/506074267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100e10 0x7fc72010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 --2- 172.21.15.26:0/506074267 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100450 0x7fc7201008d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 --2- 172.21.15.26:0/506074267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- 172.21.15.26:0/506074267 >> 172.21.15.26:0/506074267 conn(0x7fc720077410 msgr2=0x7fc7200fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- 172.21.15.26:0/506074267 shutdown_connections 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- 172.21.15.26:0/506074267 wait complete. 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 Processor -- start 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.657+0000 7fc7261a8640 1 -- start start 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 0x7fc72019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc720112270 con 0x7fc720100e10 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201120f0 con 0x7fc720106f30 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc7261a8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201123f0 con 0x7fc720100450 2024-04-23T18:20:59.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59992/0 (socket says 172.21.15.26:59992) 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 -- 172.21.15.26:0/1606921391 learned_addr learned my addr 172.21.15.26:0/1606921391 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 0x7fc72019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 -- 172.21.15.26:0/1606921391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 msgr2=0x7fc72019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 0x7fc72019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 -- 172.21.15.26:0/1606921391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 msgr2=0x7fc720198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 -- 172.21.15.26:0/1606921391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc720198ef0 con 0x7fc720100e10 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 0x7fc72019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71effd640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc714004770 tx=0x7fc7140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.658+0000 7fc71cff9640 1 -- 172.21.15.26:0/1606921391 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71403f070 con 0x7fc720100e10 2024-04-23T18:20:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.659+0000 7fc71cff9640 1 -- 172.21.15.26:0/1606921391 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71403b460 con 0x7fc720100e10 2024-04-23T18:20:59.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.659+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc720199180 con 0x7fc720100e10 2024-04-23T18:20:59.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.659+0000 7fc71cff9640 1 -- 172.21.15.26:0/1606921391 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714035070 con 0x7fc720100e10 2024-04-23T18:20:59.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.659+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7201a5310 con 0x7fc720100e10 2024-04-23T18:20:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71cff9640 1 -- 172.21.15.26:0/1606921391 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc71403b8d0 con 0x7fc720100e10 2024-04-23T18:20:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 Processor -- start 2024-04-23T18:20:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 -- start start 2024-04-23T18:20:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d85750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d94890 0x55cfe8d96c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8cea6d0 con 0x55cfe8d85c90 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8c871a0 con 0x55cfe8d85370 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc728f95740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8ce6690 con 0x55cfe8d94890 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71f7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60002/0 (socket says 172.21.15.26:60002) 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71f7fe640 1 -- 172.21.15.26:0/477708096 learned_addr learned my addr 172.21.15.26:0/477708096 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71effd640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d94890 0x55cfe8d96c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.661+0000 7fc71cff9640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 0x7fc7080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71ffff640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d85750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 -- 172.21.15.26:0/477708096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d94890 msgr2=0x55cfe8d96c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d94890 0x55cfe8d96c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 -- 172.21.15.26:0/477708096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 msgr2=0x55cfe8d85750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d85750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 -- 172.21.15.26:0/477708096 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe8c915b0 con 0x55cfe8d85c90 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71ffff640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d85750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71cff9640 1 -- 172.21.15.26:0/1606921391 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc7140f0bd0 con 0x7fc720100e10 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 0x7fc7080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc71000bd40 tx=0x7fc71000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21f5116038e7642 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc705ffb640 1 -- 172.21.15.26:0/477708096 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710015070 con 0x55cfe8d85c90 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc705ffb640 1 -- 172.21.15.26:0/477708096 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71000be30 con 0x55cfe8d85c90 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.662+0000 7fc705ffb640 1 -- 172.21.15.26:0/477708096 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100119a0 con 0x55cfe8d85c90 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- 172.21.15.26:0/477708096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 msgr2=0x55cfe8d94350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc71000bd40 tx=0x7fc71000f730 comp rx=0 tx=0).stop 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 0x7fc7080a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc718005fd0 tx=0x7fc718004a40 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- 172.21.15.26:0/477708096 shutdown_connections 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d94890 0x55cfe8d96c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d94350 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 --2- 172.21.15.26:0/477708096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d85750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- 172.21.15.26:0/477708096 >> 172.21.15.26:0/477708096 conn(0x55cfe8d817b0 msgr2=0x55cfe8d81b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- 172.21.15.26:0/477708096 shutdown_connections 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- 172.21.15.26:0/477708096 wait complete. 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.663+0000 7fc728f95740 1 -- start start 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 Processor -- start 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d94220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d91510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8d97770 con 0x55cfe8d94890 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8c84b20 con 0x55cfe8d85370 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55cfe8d978f0 con 0x55cfe8d85c90 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc728f95740 10 client.0 _ll_register_callbacks cb 0x55cfe8c84530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60008/0 (socket says 172.21.15.26:60008) 2024-04-23T18:20:59.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 -- 172.21.15.26:0/3384590697 learned_addr learned my addr 172.21.15.26:0/3384590697 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71effd640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d91510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d94220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 -- 172.21.15.26:0/3384590697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 msgr2=0x55cfe8d91510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d91510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 -- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 msgr2=0x55cfe8d94220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d94220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 -- 172.21.15.26:0/3384590697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe89fc920 con 0x55cfe8d94890 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc71ffff640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc718013950 tx=0x7fc718004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667dcf6681811881 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.664+0000 7fc6feffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71802a060 con 0x55cfe8d94890 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc71801dbe0 con 0x55cfe8d94890 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc71000be30 con 0x55cfe8d94890 2024-04-23T18:20:59.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc71801dde0 con 0x55cfe8d94890 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 10 client.16107 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cfe8ac8d30 con 0x55cfe8d94890 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71000be30 con 0x55cfe8d94890 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fc71801a4a0 con 0x55cfe8d94890 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 10 client.16107 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 10 client.16107 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 20 client.16107 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8da6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 4 client.16107 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.665+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfe8ca8500 con 0x55cfe8da4ac0 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.666+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8da6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.666+0000 7fc6feffd640 10 client.16107 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.666+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8da6f10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7364d2c5f301bc4 server_cookie=18056e1938e93a2a in_seq=0 out_seq=0 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc6feffd640 1 -- 172.21.15.26:0/3384590697 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfe8ca8500 con 0x55cfe8da4ac0 2024-04-23T18:20:59.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc6feffd640 10 client.16107 handle_command_reply: tid=0 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 client.16107 shutdown 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 msgr2=0x55cfe8d91a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc718013950 tx=0x7fc718004ab0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 shutdown_connections 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8da6f10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc6feffd640 0 client.16107 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d91a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d91510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 --2- 172.21.15.26:0/3384590697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d94220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 >> 172.21.15.26:0/3384590697 conn(0x55cfe8d817b0 msgr2=0x55cfe8d87560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 shutdown_connections 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 1 -- 172.21.15.26:0/3384590697 wait complete. 2024-04-23T18:20:59.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.667+0000 7fc728f95740 20 client.16107 trim_cache size 0 max 16384 2024-04-23T18:20:59.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 215283 5.80 MB/sec execute 54 sec latency 26332.979 ms 2024-04-23T18:20:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.684+0000 7fc728f95740 1 Processor -- start 2024-04-23T18:20:59.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.684+0000 7fc728f95740 1 -- start start 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85370 0x55cfe8da06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d94890 0x55cfe8dac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201ae2b0 con 0x55cfe8d85c90 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201abb20 con 0x55cfe8d94890 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc728f95740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201a9970 con 0x55cfe8d85370 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60020/0 (socket says 172.21.15.26:60020) 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 -- 172.21.15.26:0/32882114 learned_addr learned my addr 172.21.15.26:0/32882114 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 -- 172.21.15.26:0/32882114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85370 msgr2=0x55cfe8da06a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85370 0x55cfe8da06a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 -- 172.21.15.26:0/32882114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d94890 msgr2=0x55cfe8dac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d94890 0x55cfe8dac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 -- 172.21.15.26:0/32882114 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe8d827b0 con 0x55cfe8d85c90 2024-04-23T18:20:59.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc71effd640 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc7140ee950 tx=0x7fc7140bcaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7ffd3679957a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc7057fa640 1 -- 172.21.15.26:0/32882114 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714048070 con 0x55cfe8d85c90 2024-04-23T18:20:59.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc7057fa640 1 -- 172.21.15.26:0/32882114 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7140bd2a0 con 0x55cfe8d85c90 2024-04-23T18:20:59.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.685+0000 7fc7057fa640 1 -- 172.21.15.26:0/32882114 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7140ee5d0 con 0x55cfe8d85c90 2024-04-23T18:20:59.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- 172.21.15.26:0/32882114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 msgr2=0x55cfe8d93d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc7140ee950 tx=0x7fc7140bcaa0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- 172.21.15.26:0/32882114 shutdown_connections 2024-04-23T18:20:59.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d94890 0x55cfe8dac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d85c90 0x55cfe8d93d50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 --2- 172.21.15.26:0/32882114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85370 0x55cfe8da06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- 172.21.15.26:0/32882114 >> 172.21.15.26:0/32882114 conn(0x55cfe8d817b0 msgr2=0x55cfe8d8c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- 172.21.15.26:0/32882114 shutdown_connections 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- 172.21.15.26:0/32882114 wait complete. 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 -- start start 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.686+0000 7fc728f95740 1 Processor -- start 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d95990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d95ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201a9fe0 con 0x55cfe8d94890 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201ab450 con 0x55cfe8d85370 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201aa6b0 con 0x55cfe8d85c90 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 10 client.0 _ll_register_callbacks cb 0x7fc718013620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:60034/0 (socket says 172.21.15.26:60034) 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 -- 172.21.15.26:0/1214181259 learned_addr learned my addr 172.21.15.26:0/1214181259 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71effd640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d95ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d95990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 -- 172.21.15.26:0/1214181259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 msgr2=0x55cfe8d95ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d95ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 -- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 msgr2=0x55cfe8d95990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d95990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 -- 172.21.15.26:0/1214181259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe8d9b680 con 0x55cfe8d94890 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc71ffff640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc718013920 tx=0x7fc71801a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319f679cea1a0095 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc6feffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71801d820 con 0x55cfe8d94890 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc718004cc0 con 0x55cfe8d94890 2024-04-23T18:20:59.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7140bd2a0 con 0x55cfe8d94890 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.687+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc71802a020 con 0x55cfe8d94890 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 10 client.16113 fetch_fsmap learned FSMap version 30 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cfe8e0b7d0 con 0x55cfe8d94890 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7140bd2a0 con 0x55cfe8d94890 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fc718037070 con 0x55cfe8d94890 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 10 client.16113 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 10 client.16113 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 20 client.16113 populate_metadata read hostname 'smithi026' 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8e0df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 4 client.16113 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f715c2af", "release": true, "await": true}] 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f715c2af", "release": true, "await": true}) -- 0x55cfe8cf80f0 con 0x55cfe8da4ac0 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.688+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8e0df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.689+0000 7fc71f7fe640 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8e0df30 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b923e3f3e2088ba1 server_cookie=95056d0cda614f9 in_seq=0 out_seq=0 2024-04-23T18:20:59.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.689+0000 7fc6feffd640 10 client.16113 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:59.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc6feffd640 1 -- 172.21.15.26:0/1214181259 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55cfe8cf80f0 con 0x55cfe8da4ac0 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc6feffd640 10 client.16113 handle_command_reply: tid=0 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 client.16113 shutdown 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 msgr2=0x55cfe8d9b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc718013920 tx=0x7fc71801a640 comp rx=0 tx=0).stop 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 shutdown_connections 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55cfe8da4ac0 0x55cfe8e0df30 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55cfe8d94890 0x55cfe8d9b140 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfe8d85c90 0x55cfe8d95ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc6feffd640 0 client.16113 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 --2- 172.21.15.26:0/1214181259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfe8d85370 0x55cfe8d95990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 >> 172.21.15.26:0/1214181259 conn(0x55cfe8d817b0 msgr2=0x55cfe8d80140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 shutdown_connections 2024-04-23T18:20:59.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 1 -- 172.21.15.26:0/1214181259 wait complete. 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.696+0000 7fc728f95740 20 client.16113 trim_cache size 0 max 16384 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 30, 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 20, 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "f715c2af": { 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 20, 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:20:59.692 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 85.8, 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:20:59.693 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:20:59.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 msgr2=0x7fc7080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 0x7fc7080a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc718005fd0 tx=0x7fc718004a40 comp rx=0 tx=0).stop 2024-04-23T18:20:59.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 msgr2=0x7fc72019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:59.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc714004770 tx=0x7fc7140042f0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 shutdown_connections 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc7080a3d20 0x7fc7080a61e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc720106f30 0x7fc720198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc720100e10 0x7fc72019eab0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 --2- 172.21.15.26:0/1606921391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc720100450 0x7fc72019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 >> 172.21.15.26:0/1606921391 conn(0x7fc720077410 msgr2=0x7fc720105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 shutdown_connections 2024-04-23T18:20:59.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:20:59.697+0000 7fc7261a8640 1 -- 172.21.15.26:0/1606921391 wait complete. 2024-04-23T18:20:59.702 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f715c2af', seconds elapsed: 27.0 2024-04-23T18:20:59.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:20:59 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/681527154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:59.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:20:59 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/681527154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:20:59.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:20:59 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/681527154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:00.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 216905 5.91 MB/sec execute 55 sec latency 26357.911 ms 2024-04-23T18:21:00.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:00 smithi110 ceph-mon[32585]: pgmap v119: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:21:00.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:00 smithi026 ceph-mon[29364]: pgmap v119: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:21:00.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:00 smithi131 ceph-mon[33041]: pgmap v119: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 801 KiB/s wr, 48 op/s 2024-04-23T18:21:01.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 218090 5.90 MB/sec execute 56 sec latency 28.417 ms 2024-04-23T18:21:02.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 220430 6.04 MB/sec execute 57 sec latency 9.310 ms 2024-04-23T18:21:02.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:02 smithi110 ceph-mon[32585]: pgmap v120: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:21:02.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:02 smithi026 ceph-mon[29364]: pgmap v120: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:21:02.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:02 smithi131 ceph-mon[33041]: pgmap v120: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-04-23T18:21:03.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 222884 6.15 MB/sec execute 58 sec latency 12.776 ms 2024-04-23T18:21:04.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 224581 6.19 MB/sec execute 59 sec latency 20.967 ms 2024-04-23T18:21:04.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:04 smithi110 ceph-mon[32585]: pgmap v121: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 901 KiB/s wr, 66 op/s 2024-04-23T18:21:04.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:04 smithi026 ceph-mon[29364]: pgmap v121: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 901 KiB/s wr, 66 op/s 2024-04-23T18:21:04.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:04 smithi131 ceph-mon[33041]: pgmap v121: 97 pgs: 97 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 901 KiB/s wr, 66 op/s 2024-04-23T18:21:05.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 227326 6.37 MB/sec execute 60 sec latency 15.146 ms 2024-04-23T18:21:06.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 229096 6.36 MB/sec execute 61 sec latency 23.475 ms 2024-04-23T18:21:06.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:06 smithi110 ceph-mon[32585]: pgmap v122: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 224 op/s 2024-04-23T18:21:06.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:06 smithi026 ceph-mon[29364]: pgmap v122: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 224 op/s 2024-04-23T18:21:06.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:06 smithi131 ceph-mon[33041]: pgmap v122: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 224 op/s 2024-04-23T18:21:07.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 231193 6.48 MB/sec execute 62 sec latency 8.791 ms 2024-04-23T18:21:08.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 233778 6.58 MB/sec execute 63 sec latency 12.948 ms 2024-04-23T18:21:08.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:08 smithi110 ceph-mon[32585]: pgmap v123: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:08.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:08 smithi026 ceph-mon[29364]: pgmap v123: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:08.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:08 smithi131 ceph-mon[33041]: pgmap v123: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:09.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 235290 6.60 MB/sec execute 64 sec latency 13.049 ms 2024-04-23T18:21:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 -- 172.21.15.26:0/2743319028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 msgr2=0x7fa9f0103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 --2- 172.21.15.26:0/2743319028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0103600 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009900 tx=0x7fa9e002a480 comp rx=0 tx=0).stop 2024-04-23T18:21:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 -- 172.21.15.26:0/2743319028 shutdown_connections 2024-04-23T18:21:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 --2- 172.21.15.26:0/2743319028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 0x7fa9f010c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 --2- 172.21.15.26:0/2743319028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 0x7fa9f010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 --2- 172.21.15.26:0/2743319028 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0103600 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 -- 172.21.15.26:0/2743319028 >> 172.21.15.26:0/2743319028 conn(0x7fa9f0077fd0 msgr2=0x7fa9f00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.846+0000 7fa9f5776640 1 -- 172.21.15.26:0/2743319028 shutdown_connections 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 -- 172.21.15.26:0/2743319028 wait complete. 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 Processor -- start 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 -- start start 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 0x7fa9f0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 0x7fa9f019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112200 con 0x7fa9f0103220 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112080 con 0x7fa9f0103b40 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.847+0000 7fa9f5776640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0112380 con 0x7fa9f010c560 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59400/0 (socket says 172.21.15.26:59400) 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3129379467 learned_addr learned my addr 172.21.15.26:0/3129379467 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 0x7fa9f0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 0x7fa9f019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3129379467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 msgr2=0x7fa9f019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 0x7fa9f019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3129379467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 msgr2=0x7fa9f0198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 0x7fa9f0198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3129379467 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f019d6f0 con 0x7fa9f0103220 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005ab0 tx=0x7fa9e0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9ecff9640 1 -- 172.21.15.26:0/3129379467 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e003f070 con 0x7fa9f0103220 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f019d980 con 0x7fa9f0103220 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9ecff9640 1 -- 172.21.15.26:0/3129379467 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e0031b60 con 0x7fa9f0103220 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9ecff9640 1 -- 172.21.15.26:0/3129379467 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0035030 con 0x7fa9f0103220 2024-04-23T18:21:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.848+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f01a5270 con 0x7fa9f0103220 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.851+0000 7fa9ecff9640 1 -- 172.21.15.26:0/3129379467 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa9e0042070 con 0x7fa9f0103220 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.851+0000 7fa9f856c740 1 Processor -- start 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.851+0000 7fa9f856c740 1 -- start start 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5616ec809970 con 0x5616ec8a4e40 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5616ec7a64a0 con 0x5616ec8a4520 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9f856c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5616ec805930 con 0x5616ec8b3c10 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9ecff9640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 0x7fa9dc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.850 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:36392/0 (socket says 172.21.15.26:36392) 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9ef7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 -- 172.21.15.26:0/1310649253 learned_addr learned my addr 172.21.15.26:0/1310649253 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 0x7fa9dc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9ecff9640 1 -- 172.21.15.26:0/3129379467 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa9e00f0ec0 con 0x7fa9f0103220 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 -- 172.21.15.26:0/1310649253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 msgr2=0x5616ec8a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 -- 172.21.15.26:0/1310649253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 msgr2=0x5616ec8b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 -- 172.21.15.26:0/1310649253 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ec51c210 con 0x5616ec8b3c10 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:09.851 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.852+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009c90 tx=0x7fa9d8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43738b8c1fd04641 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 0x7fa9dc0a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005fd0 tx=0x7fa9e8009ab0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9e57fa640 1 -- 172.21.15.26:0/1310649253 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8005220 con 0x5616ec8b3c10 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9e57fa640 1 -- 172.21.15.26:0/1310649253 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9d8004c50 con 0x5616ec8b3c10 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9e57fa640 1 -- 172.21.15.26:0/1310649253 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8019670 con 0x5616ec8b3c10 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9f856c740 1 -- 172.21.15.26:0/1310649253 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 msgr2=0x5616ec8b6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.853+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009c90 tx=0x7fa9d8011810 comp rx=0 tx=0).stop 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- 172.21.15.26:0/1310649253 shutdown_connections 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8b6000 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1310649253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- 172.21.15.26:0/1310649253 >> 172.21.15.26:0/1310649253 conn(0x5616ec8a0b30 msgr2=0x5616ec8a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- 172.21.15.26:0/1310649253 shutdown_connections 2024-04-23T18:21:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- 172.21.15.26:0/1310649253 wait complete. 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- start start 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 Processor -- start 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4e40 0x5616ec8b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5616ec8b6840 con 0x5616ec8b3c10 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5616ec7a40b0 con 0x5616ec8a4520 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.854+0000 7fa9f856c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5616ec8b69c0 con 0x5616ec8a4e40 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9f856c740 10 client.0 _ll_register_callbacks cb 0x5616ec791d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4e40 0x5616ec8b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50260/0 (socket says 172.21.15.26:50260) 2024-04-23T18:21:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9ef7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 -- 172.21.15.26:0/1163273428 learned_addr learned my addr 172.21.15.26:0/1163273428 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 -- 172.21.15.26:0/1163273428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4e40 msgr2=0x5616ec8b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4e40 0x5616ec8b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 -- 172.21.15.26:0/1163273428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 msgr2=0x5616ec8b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 -- 172.21.15.26:0/1163273428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ec5e8620 con 0x5616ec8a4520 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00bcb90 tx=0x7fa9e008b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62115c94bb666d43 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0049040 con 0x5616ec8a4520 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa9e00b9500 con 0x5616ec8a4520 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.855+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d8004c50 con 0x5616ec8a4520 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa9e0037ec0 con 0x5616ec8a4520 2024-04-23T18:21:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 10 client.25583 fetch_fsmap learned FSMap version 30 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5616ec8c2700 con 0x5616ec8a4520 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9d8004c50 con 0x5616ec8a4520 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fa9e003b460 con 0x5616ec8a4520 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 10 client.25583 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 20 client.25583 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec8c3c00 0x5616ec8c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 4 client.25583 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616ec817390 con 0x5616ec8c3c00 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.856+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec8c3c00 0x5616ec8c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.857+0000 7fa9d67fc640 10 client.25583 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:09.855 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.857+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec8c3c00 0x5616ec8c5ff0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=745cc8a01d7d9dda server_cookie=95dea1b0bf1ab570 in_seq=0 out_seq=0 2024-04-23T18:21:09.856 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9d67fc640 1 -- 172.21.15.26:0/1163273428 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616ec817390 con 0x5616ec8c3c00 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9d67fc640 10 client.25583 handle_command_reply: tid=0 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 client.25583 shutdown 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 msgr2=0x5616ec8b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00bcb90 tx=0x7fa9e008b840 comp rx=0 tx=0).stop 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 shutdown_connections 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec8c3c00 0x5616ec8c5ff0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4e40 0x5616ec8b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 --2- 172.21.15.26:0/1163273428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8b34a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 >> 172.21.15.26:0/1163273428 conn(0x5616ec8a0b30 msgr2=0x5616ec8a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9d67fc640 0 client.25583 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 shutdown_connections 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 1 -- 172.21.15.26:0/1163273428 wait complete. 2024-04-23T18:21:09.857 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.858+0000 7fa9f856c740 20 client.25583 trim_cache size 0 max 16384 2024-04-23T18:21:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 Processor -- start 2024-04-23T18:21:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 -- start start 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01ae1b0 con 0x5616ec8a4e40 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01aba20 con 0x5616ec8a4520 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.874+0000 7fa9f856c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01a9870 con 0x5616ec8b3c10 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9e7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50262/0 (socket says 172.21.15.26:50262) 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3884384673 learned_addr learned my addr 172.21.15.26:0/3884384673 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3884384673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 msgr2=0x5616ec8cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3884384673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 msgr2=0x5616ec8b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 -- 172.21.15.26:0/3884384673 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ec89c2e0 con 0x5616ec8a4520 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00bce20 tx=0x7fa9e00bb3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8b44eb3a86cc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9e4ff9640 1 -- 172.21.15.26:0/3884384673 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00c5b50 con 0x5616ec8a4520 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9e4ff9640 1 -- 172.21.15.26:0/3884384673 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e00c1240 con 0x5616ec8a4520 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.875+0000 7fa9e4ff9640 1 -- 172.21.15.26:0/3884384673 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00d6150 con 0x5616ec8a4520 2024-04-23T18:21:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- 172.21.15.26:0/3884384673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 msgr2=0x5616ec8bf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00bce20 tx=0x7fa9e00bb3a0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- 172.21.15.26:0/3884384673 shutdown_connections 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8b3c10 0x5616ec8cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8a4e40 0x5616ec8b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3884384673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4520 0x5616ec8bf740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- 172.21.15.26:0/3884384673 >> 172.21.15.26:0/3884384673 conn(0x5616ec8a0b30 msgr2=0x5616ec8b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- 172.21.15.26:0/3884384673 shutdown_connections 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- 172.21.15.26:0/3884384673 wait complete. 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 -- start start 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.876+0000 7fa9f856c740 1 Processor -- start 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 0x5616ec8bbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 0x5616ec8b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01a9ee0 con 0x5616ec8b3c10 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01ab350 con 0x5616ec8a4e40 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01aa5b0 con 0x5616ec8a4520 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9f856c740 10 client.0 _ll_register_callbacks cb 0x7fa9e00bcc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:59440/0 (socket says 172.21.15.26:59440) 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 -- 172.21.15.26:0/3330772141 learned_addr learned my addr 172.21.15.26:0/3330772141 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 0x5616ec8b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 0x5616ec8bbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 -- 172.21.15.26:0/3330772141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 msgr2=0x5616ec8bbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 0x5616ec8bbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 -- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 msgr2=0x5616ec8b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 0x5616ec8b72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 -- 172.21.15.26:0/3330772141 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ec8a4c70 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9e7fff640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 0x5616ec8b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 0x5616ec8bbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9ef7fe640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8010f00 tx=0x7fa9e80099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1ca5f6dd37cc20 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.877+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8004240 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa9e80043e0 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e00c1240 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa9e80045e0 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 10 client.16131 fetch_fsmap learned FSMap version 30 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5616ec929ed0 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e00c1240 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fa9e801f070 con 0x5616ec8b3c10 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 10 client.16131 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:21:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 10 client.16131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 20 client.16131 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec92a6d0 0x5616ec92caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 4 client.16131 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616ec811200 con 0x5616ec92a6d0 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.878+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec92a6d0 0x5616ec92caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.879+0000 7fa9d67fc640 10 client.16131 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.879+0000 7fa9eeffd640 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec92a6d0 0x5616ec92caf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6989c0ae026238 server_cookie=57a1dff22a653416 in_seq=0 out_seq=0 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9d67fc640 1 -- 172.21.15.26:0/3330772141 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616ec811200 con 0x5616ec92a6d0 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9d67fc640 10 client.16131 handle_command_reply: tid=0 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 client.16131 shutdown 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 msgr2=0x5616ec8b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8010f00 tx=0x7fa9e80099c0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 shutdown_connections 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5616ec92a6d0 0x5616ec92caf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616ec8a4e40 0x5616ec8b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5616ec8b3c10 0x5616ec8b7800 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.913 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 --2- 172.21.15.26:0/3330772141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616ec8a4520 0x5616ec8bbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9d67fc640 0 client.16131 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 >> 172.21.15.26:0/3330772141 conn(0x5616ec8a0b30 msgr2=0x5616ec8ab670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 shutdown_connections 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.919+0000 7fa9f856c740 1 -- 172.21.15.26:0/3330772141 wait complete. 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f856c740 20 client.16131 trim_cache size 0 max 16384 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:21:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 msgr2=0x7fa9dc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 0x7fa9dc0a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005fd0 tx=0x7fa9e8009ab0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 msgr2=0x7fa9f0198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005ab0 tx=0x7fa9e0003c10 comp rx=0 tx=0).stop 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 shutdown_connections 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa9dc0a3dc0 0x7fa9dc0a6280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9f0103b40 0x7fa9f0198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f010c560 0x7fa9f019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.916 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 --2- 172.21.15.26:0/3129379467 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa9f0103220 0x7fa9f0198700 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:09.916 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 >> 172.21.15.26:0/3129379467 conn(0x7fa9f0077fd0 msgr2=0x7fa9f00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:09.916 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 shutdown_connections 2024-04-23T18:21:09.916 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:09.920+0000 7fa9f5776640 1 -- 172.21.15.26:0/3129379467 wait complete. 2024-04-23T18:21:10.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 237934 6.76 MB/sec execute 65 sec latency 24.407 ms 2024-04-23T18:21:10.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:10 smithi110 ceph-mon[32585]: pgmap v124: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:10.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:10 smithi026 ceph-mon[29364]: pgmap v124: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:10.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:10 smithi131 ceph-mon[33041]: pgmap v124: 97 pgs: 97 active+clean; 113 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-04-23T18:21:11.667 INFO:tasks.workunit.client.0.smithi026.stdout: 1 239557 6.75 MB/sec execute 66 sec latency 15.278 ms 2024-04-23T18:21:12.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 241722 6.85 MB/sec execute 67 sec latency 14.061 ms 2024-04-23T18:21:12.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:12 smithi110 ceph-mon[32585]: pgmap v125: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 346 op/s 2024-04-23T18:21:12.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:12 smithi026 ceph-mon[29364]: pgmap v125: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 346 op/s 2024-04-23T18:21:12.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:12 smithi131 ceph-mon[33041]: pgmap v125: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 346 op/s 2024-04-23T18:21:13.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 244158 6.93 MB/sec execute 68 sec latency 15.675 ms 2024-04-23T18:21:14.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 245660 6.94 MB/sec execute 69 sec latency 18.658 ms 2024-04-23T18:21:14.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:14 smithi110 ceph-mon[32585]: pgmap v126: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-04-23T18:21:14.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:14 smithi026 ceph-mon[29364]: pgmap v126: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-04-23T18:21:14.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:14 smithi131 ceph-mon[33041]: pgmap v126: 97 pgs: 97 active+clean; 119 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-04-23T18:21:15.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 248448 7.08 MB/sec execute 70 sec latency 10.869 ms 2024-04-23T18:21:16.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 249764 7.07 MB/sec execute 71 sec latency 15.589 ms 2024-04-23T18:21:16.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:16 smithi110 ceph-mon[32585]: pgmap v127: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-04-23T18:21:16.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:16 smithi026 ceph-mon[29364]: pgmap v127: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-04-23T18:21:16.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:16 smithi131 ceph-mon[33041]: pgmap v127: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-04-23T18:21:17.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 252239 7.17 MB/sec execute 72 sec latency 9.564 ms 2024-04-23T18:21:18.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 253795 7.15 MB/sec execute 73 sec latency 15.048 ms 2024-04-23T18:21:18.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:18 smithi110 ceph-mon[32585]: pgmap v128: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:18.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:18 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:18.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:18 smithi026 ceph-mon[29364]: pgmap v128: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:18.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:18 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:18.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:18 smithi131 ceph-mon[33041]: pgmap v128: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:18.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:18 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:19.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 255886 7.23 MB/sec execute 74 sec latency 11.328 ms 2024-04-23T18:21:19.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:19.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:19.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:19 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:21:19.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:19.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:19.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:19 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:21:19.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:19.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:19.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:19 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:21:20.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 257900 7.24 MB/sec execute 75 sec latency 14.649 ms 2024-04-23T18:21:21.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:20 smithi110 ceph-mon[32585]: pgmap v129: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:21.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:20 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:20 smithi026 ceph-mon[29364]: pgmap v129: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:20 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:20 smithi131 ceph-mon[33041]: pgmap v129: 97 pgs: 97 active+clean; 129 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-23T18:21:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:20 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:21.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 259631 7.30 MB/sec execute 76 sec latency 18.074 ms 2024-04-23T18:21:22.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 262343 7.39 MB/sec execute 77 sec latency 11.878 ms 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 -- 172.21.15.26:0/856733992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe954102b80 msgr2=0x7fe954106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 --2- 172.21.15.26:0/856733992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe954102b80 0x7fe954106bc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe940009900 tx=0x7fe94002a410 comp rx=0 tx=0).stop 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 -- 172.21.15.26:0/856733992 shutdown_connections 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 --2- 172.21.15.26:0/856733992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 0x7fe95410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 --2- 172.21.15.26:0/856733992 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe954102b80 0x7fe954106bc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 --2- 172.21.15.26:0/856733992 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 0x7fe9541025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.012+0000 7fe95ac60640 1 -- 172.21.15.26:0/856733992 >> 172.21.15.26:0/856733992 conn(0x7fe9540fda10 msgr2=0x7fe9540ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- 172.21.15.26:0/856733992 shutdown_connections 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- 172.21.15.26:0/856733992 wait complete. 2024-04-23T18:21:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 Processor -- start 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- start start 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 0x7fe954198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 0x7fe954198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541121e0 con 0x7fe95419cf90 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe954112060 con 0x7fe9541072a0 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe95ac60640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe954112360 con 0x7fe9541021d0 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe9591fd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe9591fd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53330/0 (socket says 172.21.15.26:53330) 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe9591fd640 1 -- 172.21.15.26:0/968736966 learned_addr learned my addr 172.21.15.26:0/968736966 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.013+0000 7fe9589fc640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 0x7fe954198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe953fff640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 0x7fe954198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 -- 172.21.15.26:0/968736966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 msgr2=0x7fe954198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 0x7fe954198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 -- 172.21.15.26:0/968736966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 msgr2=0x7fe954198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 0x7fe954198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 -- 172.21.15.26:0/968736966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95419da80 con 0x7fe95419cf90 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe9591fd640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00eb10 tx=0x7fe94c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe951ffb640 1 -- 172.21.15.26:0/968736966 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c004b20 con 0x7fe95419cf90 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe951ffb640 1 -- 172.21.15.26:0/968736966 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe94c004cc0 con 0x7fe95419cf90 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95419dd70 con 0x7fe95419cf90 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe951ffb640 1 -- 172.21.15.26:0/968736966 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c010670 con 0x7fe95419cf90 2024-04-23T18:21:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.014+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9541a54e0 con 0x7fe95419cf90 2024-04-23T18:21:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.016+0000 7fe951ffb640 1 -- 172.21.15.26:0/968736966 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe94c005020 con 0x7fe95419cf90 2024-04-23T18:21:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe951ffb640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 0x7fe9440a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 Processor -- start 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 -- start start 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe951ffb640 1 -- 172.21.15.26:0/968736966 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe94c0d0cd0 con 0x7fe95419cf90 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe9589fc640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 0x7fe9440a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f96a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f97c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56497f8cf970 con 0x56497f96ae20 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56497f86c4a0 con 0x56497f96a500 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe95d975740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56497f8cb930 con 0x56497f979c10 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe953fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f96a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe953fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53332/0 (socket says 172.21.15.26:53332) 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe953fff640 1 -- 172.21.15.26:0/2810891085 learned_addr learned my addr 172.21.15.26:0/2810891085 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.017+0000 7fe9591fd640 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f97c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 -- 172.21.15.26:0/2810891085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 msgr2=0x56497f97c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f97c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe9589fc640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 0x7fe9440a6280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe948011550 tx=0x7fe948011950 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 -- 172.21.15.26:0/2810891085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 msgr2=0x56497f96a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f96a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 -- 172.21.15.26:0/2810891085 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56497f5e2210 con 0x56497f96ae20 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe9589fc640 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f96a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe953fff640 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe940033000 tx=0x7fe940005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46210a712de62632 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe93effd640 1 -- 172.21.15.26:0/2810891085 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940035040 con 0x56497f96ae20 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe93effd640 1 -- 172.21.15.26:0/2810891085 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe940004030 con 0x56497f96ae20 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.018+0000 7fe93effd640 1 -- 172.21.15.26:0/2810891085 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940005380 con 0x56497f96ae20 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- 172.21.15.26:0/2810891085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 msgr2=0x56497f9796d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe940033000 tx=0x7fe940005e40 comp rx=0 tx=0).stop 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- 172.21.15.26:0/2810891085 shutdown_connections 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f97c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96ae20 0x56497f9796d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 --2- 172.21.15.26:0/2810891085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f96a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- 172.21.15.26:0/2810891085 >> 172.21.15.26:0/2810891085 conn(0x56497f966b30 msgr2=0x56497f966f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- 172.21.15.26:0/2810891085 shutdown_connections 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- 172.21.15.26:0/2810891085 wait complete. 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 -- start start 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.019+0000 7fe95d975740 1 Processor -- start 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f9795a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f976890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56497f97c7e0 con 0x56497f979c10 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56497f86a0b0 con 0x56497f96a500 2024-04-23T18:21:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56497f97c960 con 0x56497f96ae20 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe95d975740 10 client.0 _ll_register_callbacks cb 0x56497f857d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe9591fd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f976890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe9591fd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53336/0 (socket says 172.21.15.26:53336) 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f9795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe9591fd640 1 -- 172.21.15.26:0/3585772769 learned_addr learned my addr 172.21.15.26:0/3585772769 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 -- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 msgr2=0x56497f9795a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f9795a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 -- 172.21.15.26:0/3585772769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 msgr2=0x56497f976e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 -- 172.21.15.26:0/3585772769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56497f6ae620 con 0x56497f96ae20 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe9591fd640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe953fff640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f976890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe94002a930 tx=0x7fe94002ab80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8a118aeae362eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe933fff640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.020+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940052060 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe9400043f0 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe940004030 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe940035070 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 10 client.25585 fetch_fsmap learned FSMap version 31 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56497f97c7e0 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe940004030 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fe9400047e0 con 0x56497f96ae20 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 10 client.25585 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 10 client.25585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 20 client.25585 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9899e0 0x56497f98bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 4 client.25585 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56497f8dd390 con 0x56497f9899e0 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.021+0000 7fe9589fc640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9899e0 0x56497f98bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe933fff640 10 client.25585 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe9589fc640 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9899e0 0x56497f98bdd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f57462b0022864c9 server_cookie=b5c80a37e91063d7 in_seq=0 out_seq=0 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe933fff640 1 -- 172.21.15.26:0/3585772769 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56497f8dd390 con 0x56497f9899e0 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe933fff640 10 client.25585 handle_command_reply: tid=0 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe95d975740 1 client.25585 shutdown 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 msgr2=0x56497f976890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.022+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f976890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe94002a930 tx=0x7fe94002ab80 comp rx=0 tx=0).stop 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 shutdown_connections 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9899e0 0x56497f98bdd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f979c10 0x56497f976e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f976890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 --2- 172.21.15.26:0/3585772769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96a500 0x56497f9795a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe933fff640 0 client.25585 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 >> 172.21.15.26:0/3585772769 conn(0x56497f966b30 msgr2=0x56497f96c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 shutdown_connections 2024-04-23T18:21:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 1 -- 172.21.15.26:0/3585772769 wait complete. 2024-04-23T18:21:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.023+0000 7fe95d975740 20 client.25585 trim_cache size 0 max 16384 2024-04-23T18:21:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 Processor -- start 2024-04-23T18:21:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 -- start start 2024-04-23T18:21:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96ae20 0x56497f9790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f9914f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541ae090 con 0x56497f96a500 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541ab960 con 0x56497f96ae20 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.039+0000 7fe95d975740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541a97b0 con 0x56497f979c10 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53344/0 (socket says 172.21.15.26:53344) 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 -- 172.21.15.26:0/368519985 learned_addr learned my addr 172.21.15.26:0/368519985 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe953fff640 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96ae20 0x56497f9790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9591fd640 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f9914f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 -- 172.21.15.26:0/368519985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 msgr2=0x56497f9914f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f9914f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 -- 172.21.15.26:0/368519985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96ae20 msgr2=0x56497f9790d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96ae20 0x56497f9790d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 -- 172.21.15.26:0/368519985 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56497f976c00 con 0x56497f96a500 2024-04-23T18:21:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe9589fc640 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe9480042c0 tx=0x7fe948020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cef3945a7777ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe93e7fc640 1 -- 172.21.15.26:0/368519985 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948032070 con 0x56497f96a500 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe93e7fc640 1 -- 172.21.15.26:0/368519985 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe94800fe40 con 0x56497f96a500 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.040+0000 7fe93e7fc640 1 -- 172.21.15.26:0/368519985 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94801a040 con 0x56497f96a500 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- 172.21.15.26:0/368519985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 msgr2=0x56497f9856e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe9480042c0 tx=0x7fe948020000 comp rx=0 tx=0).stop 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- 172.21.15.26:0/368519985 shutdown_connections 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f96ae20 0x56497f9790d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f979c10 0x56497f9914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 --2- 172.21.15.26:0/368519985 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f9856e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- 172.21.15.26:0/368519985 >> 172.21.15.26:0/368519985 conn(0x56497f966b30 msgr2=0x56497f966f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- 172.21.15.26:0/368519985 shutdown_connections 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- 172.21.15.26:0/368519985 wait complete. 2024-04-23T18:21:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 -- start start 2024-04-23T18:21:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.041+0000 7fe95d975740 1 Processor -- start 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f982130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f979c10 0x56497f972cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541a9e20 con 0x56497f96a500 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541ab290 con 0x56497f979c10 2024-04-23T18:21:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe9541aa4f0 con 0x56497f96ae20 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 10 client.0 _ll_register_callbacks cb 0x7fe940005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53350/0 (socket says 172.21.15.26:53350) 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 -- 172.21.15.26:0/431184465 learned_addr learned my addr 172.21.15.26:0/431184465 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe953fff640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f982130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9591fd640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f979c10 0x56497f972cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 -- 172.21.15.26:0/431184465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 msgr2=0x56497f982130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f982130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 -- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f979c10 msgr2=0x56497f972cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f979c10 0x56497f972cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 -- 172.21.15.26:0/431184465 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56497f973430 con 0x56497f96a500 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe953fff640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f982130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe9589fc640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe948009b70 tx=0x7fe94802ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46585d730d5e2aff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe933fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948032070 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe948005040 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.042+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56497f8d7200 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe94801a070 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 10 client.16149 fetch_fsmap learned FSMap version 31 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56497f9ef7d0 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56497f8d7200 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fe94801d030 con 0x56497f96a500 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 10 client.16149 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 10 client.16149 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 20 client.16149 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9efe40 0x56497f9f2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 4 client.16149 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56497f8535f0 con 0x56497f9efe40 2024-04-23T18:21:23.042 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe953fff640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9efe40 0x56497f9f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.042 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe933fff640 10 client.16149 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:23.042 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.043+0000 7fe953fff640 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9efe40 0x56497f9f2260 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe94b76830db54fb server_cookie=97afddfcd7710945 in_seq=0 out_seq=0 2024-04-23T18:21:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe933fff640 1 -- 172.21.15.26:0/431184465 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56497f8535f0 con 0x56497f9efe40 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe933fff640 10 client.16149 handle_command_reply: tid=0 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 client.16149 shutdown 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 msgr2=0x56497f981bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe948009b70 tx=0x7fe94802ca40 comp rx=0 tx=0).stop 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 shutdown_connections 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56497f9efe40 0x56497f9f2260 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56497f979c10 0x56497f972cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56497f96ae20 0x56497f982130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 --2- 172.21.15.26:0/431184465 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56497f96a500 0x56497f981bf0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe933fff640 0 client.16149 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 >> 172.21.15.26:0/431184465 conn(0x56497f966b30 msgr2=0x56497f965420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 shutdown_connections 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 1 -- 172.21.15.26:0/431184465 wait complete. 2024-04-23T18:21:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.125+0000 7fe95d975740 20 client.16149 trim_cache size 0 max 16384 2024-04-23T18:21:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:21:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:21:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:21:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 msgr2=0x7fe9440a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 0x7fe9440a6280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe948011550 tx=0x7fe948011950 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 msgr2=0x7fe95419d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00eb10 tx=0x7fe94c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 shutdown_connections 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe9440a3dc0 0x7fe9440a6280 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9541072a0 0x7fe954198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe95419cf90 0x7fe95419d440 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 --2- 172.21.15.26:0/968736966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9541021d0 0x7fe954198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 >> 172.21.15.26:0/968736966 conn(0x7fe9540fda10 msgr2=0x7fe9540fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 shutdown_connections 2024-04-23T18:21:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:23.126+0000 7fe95ac60640 1 -- 172.21.15.26:0/968736966 wait complete. 2024-04-23T18:21:23.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:22 smithi110 ceph-mon[32585]: pgmap v130: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 341 op/s 2024-04-23T18:21:23.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:22 smithi026 ceph-mon[29364]: pgmap v130: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 341 op/s 2024-04-23T18:21:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:22 smithi131 ceph-mon[33041]: pgmap v130: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 341 op/s 2024-04-23T18:21:23.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 263581 7.38 MB/sec execute 78 sec latency 56.240 ms 2024-04-23T18:21:24.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 266202 7.49 MB/sec execute 79 sec latency 10.680 ms 2024-04-23T18:21:25.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:24 smithi110 ceph-mon[32585]: pgmap v131: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-23T18:21:25.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:24 smithi026 ceph-mon[29364]: pgmap v131: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-23T18:21:25.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:24 smithi131 ceph-mon[33041]: pgmap v131: 97 pgs: 97 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 201 op/s 2024-04-23T18:21:25.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 267513 7.47 MB/sec execute 80 sec latency 18.159 ms 2024-04-23T18:21:26.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 269914 7.55 MB/sec execute 81 sec latency 12.022 ms 2024-04-23T18:21:27.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:26 smithi110 ceph-mon[32585]: pgmap v132: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 329 op/s 2024-04-23T18:21:27.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:26 smithi026 ceph-mon[29364]: pgmap v132: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 329 op/s 2024-04-23T18:21:27.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:26 smithi131 ceph-mon[33041]: pgmap v132: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 329 op/s 2024-04-23T18:21:27.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 271803 7.53 MB/sec execute 82 sec latency 14.169 ms 2024-04-23T18:21:28.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 273703 7.60 MB/sec execute 83 sec latency 11.391 ms 2024-04-23T18:21:29.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:28 smithi110 ceph-mon[32585]: pgmap v133: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:28 smithi026 ceph-mon[29364]: pgmap v133: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:28 smithi131 ceph-mon[33041]: pgmap v133: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:29.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 275553 7.60 MB/sec execute 84 sec latency 191.267 ms 2024-04-23T18:21:30.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 277267 7.65 MB/sec execute 85 sec latency 11.363 ms 2024-04-23T18:21:31.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:30 smithi110 ceph-mon[32585]: pgmap v134: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:30 smithi026 ceph-mon[29364]: pgmap v134: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:31.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:30 smithi131 ceph-mon[33041]: pgmap v134: 97 pgs: 97 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:21:31.668 INFO:tasks.workunit.client.0.smithi026.stdout: 1 279625 7.71 MB/sec execute 86 sec latency 17.220 ms 2024-04-23T18:21:32.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 281221 7.70 MB/sec execute 87 sec latency 15.112 ms 2024-04-23T18:21:33.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:32 smithi110 ceph-mon[32585]: pgmap v135: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:21:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:32 smithi026 ceph-mon[29364]: pgmap v135: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:21:33.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:32 smithi131 ceph-mon[33041]: pgmap v135: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:21:33.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 284028 7.82 MB/sec execute 88 sec latency 14.905 ms 2024-04-23T18:21:34.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 285548 7.79 MB/sec execute 89 sec latency 13.903 ms 2024-04-23T18:21:35.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:34 smithi110 ceph-mon[32585]: pgmap v136: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 220 op/s 2024-04-23T18:21:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:34 smithi026 ceph-mon[29364]: pgmap v136: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 220 op/s 2024-04-23T18:21:35.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:34 smithi131 ceph-mon[33041]: pgmap v136: 97 pgs: 97 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 220 op/s 2024-04-23T18:21:35.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 287775 7.86 MB/sec execute 90 sec latency 11.666 ms 2024-04-23T18:21:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 -- 172.21.15.26:0/1725731629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 msgr2=0x7f159c10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 --2- 172.21.15.26:0/1725731629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c10bfe0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1590009900 tx=0x7f159002a510 comp rx=0 tx=0).stop 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 -- 172.21.15.26:0/1725731629 shutdown_connections 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 --2- 172.21.15.26:0/1725731629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c10c520 0x7f159c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 --2- 172.21.15.26:0/1725731629 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c10bfe0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 --2- 172.21.15.26:0/1725731629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c1031e0 0x7f159c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 -- 172.21.15.26:0/1725731629 >> 172.21.15.26:0/1725731629 conn(0x7f159c077fd0 msgr2=0x7f159c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 -- 172.21.15.26:0/1725731629 shutdown_connections 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.219+0000 7f15a19af640 1 -- 172.21.15.26:0/1725731629 wait complete. 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 Processor -- start 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 -- start start 2024-04-23T18:21:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c1031e0 0x7f159c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c10c520 0x7f159c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1121b0 con 0x7f159c103b00 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f159c112030 con 0x7f159c10c520 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f15a19af640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f159c112330 con 0x7f159c1031e0 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f159a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f159affd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c1031e0 0x7f159c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f159a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46510/0 (socket says 172.21.15.26:46510) 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f159a7fc640 1 -- 172.21.15.26:0/1930224496 learned_addr learned my addr 172.21.15.26:0/1930224496 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.220+0000 7f159b7fe640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c10c520 0x7f159c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 -- 172.21.15.26:0/1930224496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c1031e0 msgr2=0x7f159c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c1031e0 0x7f159c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 -- 172.21.15.26:0/1930224496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c10c520 msgr2=0x7f159c19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c10c520 0x7f159c19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 -- 172.21.15.26:0/1930224496 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c19d6b0 con 0x7f159c103b00 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f159a7fc640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1590005940 tx=0x7f15900040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f15a09ad640 1 -- 172.21.15.26:0/1930224496 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159003f030 con 0x7f159c103b00 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f15a09ad640 1 -- 172.21.15.26:0/1930224496 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15900044c0 con 0x7f159c103b00 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c19d940 con 0x7f159c103b00 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f15a09ad640 1 -- 172.21.15.26:0/1930224496 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1590035070 con 0x7f159c103b00 2024-04-23T18:21:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.221+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c1a5230 con 0x7f159c103b00 2024-04-23T18:21:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.223+0000 7f15a09ad640 1 -- 172.21.15.26:0/1930224496 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1590042030 con 0x7f159c103b00 2024-04-23T18:21:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 Processor -- start 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 -- start start 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf2e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a09ad640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 0x7f15840a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2ee20 0x56154cf3d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56154ce93970 con 0x56154cf3dc10 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56154ce304a0 con 0x56154cf2e500 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.224+0000 7f15a479c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56154ce8f930 con 0x56154cf2ee20 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159affd640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 0x7f15840a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46520/0 (socket says 172.21.15.26:46520) 2024-04-23T18:21:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 -- 172.21.15.26:0/782243693 learned_addr learned my addr 172.21.15.26:0/782243693 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 -- 172.21.15.26:0/782243693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2ee20 msgr2=0x56154cf3d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159a7fc640 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2ee20 0x56154cf3d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2ee20 0x56154cf3d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159affd640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 0x7f15840a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f158c0095f0 tx=0x7f158c005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 -- 172.21.15.26:0/782243693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 msgr2=0x56154cf2e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f15a09ad640 1 -- 172.21.15.26:0/1930224496 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f15900f0a40 con 0x7f159c103b00 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf2e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 -- 172.21.15.26:0/782243693 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154cba6210 con 0x56154cf3dc10 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f159b7fe640 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1594009f90 tx=0x7f15940119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7600581a4adc03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f1581ffb640 1 -- 172.21.15.26:0/782243693 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594004d20 con 0x56154cf3dc10 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f1581ffb640 1 -- 172.21.15.26:0/782243693 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1594003e20 con 0x56154cf3dc10 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.225+0000 7f1581ffb640 1 -- 172.21.15.26:0/782243693 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159400f740 con 0x56154cf3dc10 2024-04-23T18:21:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 -- 172.21.15.26:0/782243693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 msgr2=0x56154cf40000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1594009f90 tx=0x7f15940119d0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 -- 172.21.15.26:0/782243693 shutdown_connections 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf3dc10 0x56154cf40000 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2ee20 0x56154cf3d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 --2- 172.21.15.26:0/782243693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf2e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 -- 172.21.15.26:0/782243693 >> 172.21.15.26:0/782243693 conn(0x56154cf2ab30 msgr2=0x56154cf2af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 -- 172.21.15.26:0/782243693 shutdown_connections 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.226+0000 7f15a479c740 1 -- 172.21.15.26:0/782243693 wait complete. 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 -- start start 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 Processor -- start 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2e500 0x56154cf3d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf3dc10 0x56154cf3ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56154cf40840 con 0x56154cf2ee20 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56154ce2e0b0 con 0x56154cf3dc10 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56154cf409c0 con 0x56154cf2e500 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f15a479c740 10 client.0 _ll_register_callbacks cb 0x56154ce1bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46524/0 (socket says 172.21.15.26:46524) 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 -- 172.21.15.26:0/800236413 learned_addr learned my addr 172.21.15.26:0/800236413 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159affd640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2e500 0x56154cf3d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159b7fe640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf3dc10 0x56154cf3ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 -- 172.21.15.26:0/800236413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2e500 msgr2=0x56154cf3d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2e500 0x56154cf3d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 -- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf3dc10 msgr2=0x56154cf3ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.227+0000 7f159a7fc640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf3dc10 0x56154cf3ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f159a7fc640 1 -- 172.21.15.26:0/800236413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154cc72620 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f159a7fc640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f15900c5150 tx=0x7f15900bc6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8653113a55d5fe17 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1590049070 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f15900b9130 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1594003e20 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f15900c0dc0 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 10 client.16161 fetch_fsmap learned FSMap version 32 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56154cf4c890 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1594003e20 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 32) ==== 8342+0+0 (secure 0 0 0) 0x7f15900c51f0 con 0x56154cf2ee20 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 10 client.16161 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 10 client.16161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 20 client.16161 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cf4dcf0 0x56154cf50120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 4 client.16161 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.228+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56154cea1390 con 0x56154cf4dcf0 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.229+0000 7f159affd640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cf4dcf0 0x56154cf50120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.229+0000 7f159affd640 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cf4dcf0 0x56154cf50120 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa5a6a9a22b3b319 server_cookie=523495302b8c0acb in_seq=0 out_seq=0 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.229+0000 7f157affd640 10 client.16161 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f157affd640 1 -- 172.21.15.26:0/800236413 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56154cea1390 con 0x56154cf4dcf0 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f157affd640 10 client.16161 handle_command_reply: tid=0 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 client.16161 shutdown 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 msgr2=0x56154cf3a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f15900c5150 tx=0x7f15900bc6c0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 shutdown_connections 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cf4dcf0 0x56154cf50120 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf3dc10 0x56154cf3ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf3a8b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f157affd640 0 client.16161 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 --2- 172.21.15.26:0/800236413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf2e500 0x56154cf3d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 >> 172.21.15.26:0/800236413 conn(0x56154cf2ab30 msgr2=0x56154cf309a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 shutdown_connections 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 1 -- 172.21.15.26:0/800236413 wait complete. 2024-04-23T18:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.230+0000 7f15a479c740 20 client.16161 trim_cache size 0 max 16384 2024-04-23T18:21:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 Processor -- start 2024-04-23T18:21:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 -- start start 2024-04-23T18:21:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2ee20 0x56154cf3d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cf55a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1ae0e0 con 0x56154cf2e500 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1ab980 con 0x56154cf2ee20 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f15a479c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1a97d0 con 0x56154cf3dc10 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:46530/0 (socket says 172.21.15.26:46530) 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 -- 172.21.15.26:0/2407349474 learned_addr learned my addr 172.21.15.26:0/2407349474 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159b7fe640 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cf55a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159a7fc640 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2ee20 0x56154cf3d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 -- 172.21.15.26:0/2407349474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 msgr2=0x56154cf55a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cf55a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 -- 172.21.15.26:0/2407349474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2ee20 msgr2=0x56154cf3d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2ee20 0x56154cf3d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.246+0000 7f159affd640 1 -- 172.21.15.26:0/2407349474 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154cf507a0 con 0x56154cf2e500 2024-04-23T18:21:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.247+0000 7f159affd640 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f158c004550 tx=0x7f158c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc00bc72d8a8b3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.247+0000 7f15817fa640 1 -- 172.21.15.26:0/2407349474 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c02d070 con 0x56154cf2e500 2024-04-23T18:21:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.247+0000 7f15817fa640 1 -- 172.21.15.26:0/2407349474 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f158c0272d0 con 0x56154cf2e500 2024-04-23T18:21:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.247+0000 7f15817fa640 1 -- 172.21.15.26:0/2407349474 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c01e6c0 con 0x56154cf2e500 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- 172.21.15.26:0/2407349474 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 msgr2=0x56154cf497a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f158c004550 tx=0x7f158c014000 comp rx=0 tx=0).stop 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- 172.21.15.26:0/2407349474 shutdown_connections 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2ee20 0x56154cf3d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cf55a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 --2- 172.21.15.26:0/2407349474 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2e500 0x56154cf497a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- 172.21.15.26:0/2407349474 >> 172.21.15.26:0/2407349474 conn(0x56154cf2ab30 msgr2=0x56154cf35f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- 172.21.15.26:0/2407349474 shutdown_connections 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- 172.21.15.26:0/2407349474 wait complete. 2024-04-23T18:21:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 -- start start 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.248+0000 7f15a479c740 1 Processor -- start 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf37eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cfb32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1a9e40 con 0x56154cf2ee20 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1ab2b0 con 0x56154cf2e500 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1aa510 con 0x56154cf3dc10 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f15a479c740 10 client.0 _ll_register_callbacks cb 0x7f15900bc890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159b7fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cfb32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35548/0 (socket says 172.21.15.26:35548) 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 -- 172.21.15.26:0/3588440448 learned_addr learned my addr 172.21.15.26:0/3588440448 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:36.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159a7fc640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf37eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 -- 172.21.15.26:0/3588440448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 msgr2=0x56154cfb32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cfb32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 -- 172.21.15.26:0/3588440448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 msgr2=0x56154cf37eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf37eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 -- 172.21.15.26:0/3588440448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154cf40000 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f159affd640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f158c0118d0 tx=0x7f158c003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6130beec89cbe1cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.249+0000 7f157affd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c02d070 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f158c027790 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f158c0272d0 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f158c027990 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 10 client.25622 fetch_fsmap learned FSMap version 32 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56154cf2fb70 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f158c0272d0 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8342+0+0 (secure 0 0 0) 0x7f158c004c50 con 0x56154cf2e500 2024-04-23T18:21:36.247 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 10 client.25622 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 10 client.25622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 20 client.25622 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cfb4ce0 0x56154cfb7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 4 client.25622 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56154ce9b200 con 0x56154cfb4ce0 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.250+0000 7f159a7fc640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cfb4ce0 0x56154cfb7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.251+0000 7f157affd640 10 client.25622 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:36.248 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.251+0000 7f159a7fc640 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cfb4ce0 0x56154cfb7100 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=898750757f4bbb14 server_cookie=e7ab757034ac0225 in_seq=0 out_seq=0 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.291+0000 7f157affd640 1 -- 172.21.15.26:0/3588440448 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56154ce9b200 con 0x56154cfb4ce0 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.291+0000 7f157affd640 10 client.25622 handle_command_reply: tid=0 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.291+0000 7f15a479c740 1 client.25622 shutdown 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 msgr2=0x56154cf37970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f158c0118d0 tx=0x7f158c003f30 comp rx=0 tx=0).stop 2024-04-23T18:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 shutdown_connections 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56154cfb4ce0 0x56154cfb7100 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56154cf3dc10 0x56154cfb32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f157affd640 0 client.25622 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56154cf2ee20 0x56154cf37eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 --2- 172.21.15.26:0/3588440448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154cf2e500 0x56154cf37970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 >> 172.21.15.26:0/3588440448 conn(0x56154cf2ab30 msgr2=0x56154cf29420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 shutdown_connections 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 1 -- 172.21.15.26:0/3588440448 wait complete. 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.292+0000 7f15a479c740 20 client.25622 trim_cache size 0 max 16384 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 msgr2=0x7f15840a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 0x7f15840a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f158c0095f0 tx=0x7f158c005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 msgr2=0x7f159c198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1590005940 tx=0x7f15900040c0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 shutdown_connections 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f15840a3e40 0x7f15840a6300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159c10c520 0x7f159c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f159c103b00 0x7f159c198c50 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 --2- 172.21.15.26:0/1930224496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f159c1031e0 0x7f159c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 >> 172.21.15.26:0/1930224496 conn(0x7f159c077fd0 msgr2=0x7f159c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 shutdown_connections 2024-04-23T18:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:36.293+0000 7f15a19af640 1 -- 172.21.15.26:0/1930224496 wait complete. 2024-04-23T18:21:36.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 289657 7.85 MB/sec execute 91 sec latency 51.217 ms 2024-04-23T18:21:37.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:36 smithi110 ceph-mon[32585]: pgmap v137: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T18:21:37.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:36 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:36 smithi026 ceph-mon[29364]: pgmap v137: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T18:21:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:36 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:37.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:36 smithi131 ceph-mon[33041]: pgmap v137: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T18:21:37.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:36 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:37.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 291416 7.90 MB/sec execute 92 sec latency 18.724 ms 2024-04-23T18:21:38.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 293856 7.95 MB/sec execute 93 sec latency 18.641 ms 2024-04-23T18:21:39.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:38 smithi110 ceph-mon[32585]: pgmap v138: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:39.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:38 smithi026 ceph-mon[29364]: pgmap v138: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:39.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:38 smithi131 ceph-mon[33041]: pgmap v138: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:39.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 295337 7.94 MB/sec execute 94 sec latency 17.906 ms 2024-04-23T18:21:40.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 298083 8.04 MB/sec execute 95 sec latency 9.675 ms 2024-04-23T18:21:41.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:40 smithi110 ceph-mon[32585]: pgmap v139: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:40 smithi026 ceph-mon[29364]: pgmap v139: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:41.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:40 smithi131 ceph-mon[33041]: pgmap v139: 97 pgs: 97 active+clean; 156 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-23T18:21:41.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 299194 7.99 MB/sec execute 96 sec latency 41.983 ms 2024-04-23T18:21:42.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:41 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:42.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:41 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:42.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:41 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:21:42.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 301791 8.08 MB/sec execute 97 sec latency 51.318 ms 2024-04-23T18:21:43.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:42 smithi110 ceph-mon[32585]: pgmap v140: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-04-23T18:21:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:42 smithi026 ceph-mon[29364]: pgmap v140: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-04-23T18:21:43.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:42 smithi131 ceph-mon[33041]: pgmap v140: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-04-23T18:21:43.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 303333 8.05 MB/sec execute 98 sec latency 14.872 ms 2024-04-23T18:21:44.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 305499 8.11 MB/sec execute 99 sec latency 57.908 ms 2024-04-23T18:21:45.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:44 smithi110 ceph-mon[32585]: pgmap v141: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 255 op/s 2024-04-23T18:21:45.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:44 smithi026 ceph-mon[29364]: pgmap v141: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 255 op/s 2024-04-23T18:21:45.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:44 smithi131 ceph-mon[33041]: pgmap v141: 97 pgs: 97 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 255 op/s 2024-04-23T18:21:45.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 307627 8.11 MB/sec execute 100 sec latency 23.524 ms 2024-04-23T18:21:46.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 309334 8.14 MB/sec execute 101 sec latency 14.697 ms 2024-04-23T18:21:47.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:46 smithi110 ceph-mon[32585]: pgmap v142: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-23T18:21:47.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:46 smithi026 ceph-mon[29364]: pgmap v142: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-23T18:21:47.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:46 smithi131 ceph-mon[33041]: pgmap v142: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-23T18:21:47.617 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:21:47.617 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.1 2024-04-23T18:21:47.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:21:47.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 311962 8.20 MB/sec execute 102 sec latency 47.492 ms 2024-04-23T18:21:47.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 -- 172.21.15.26:0/2907227168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c102bf0 msgr2=0x7f287c106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:47.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 --2- 172.21.15.26:0/2907227168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c102bf0 0x7f287c106ba0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2870009900 tx=0x7f287002a410 comp rx=0 tx=0).stop 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 -- 172.21.15.26:0/2907227168 shutdown_connections 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 --2- 172.21.15.26:0/2907227168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 --2- 172.21.15.26:0/2907227168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c102bf0 0x7f287c106ba0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 --2- 172.21.15.26:0/2907227168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 0x7f287c102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 -- 172.21.15.26:0/2907227168 >> 172.21.15.26:0/2907227168 conn(0x7f287c0790e0 msgr2=0x7f287c0794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 -- 172.21.15.26:0/2907227168 shutdown_connections 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.737+0000 7f2882b37640 1 -- 172.21.15.26:0/2907227168 wait complete. 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 Processor -- start 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 -- start start 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 0x7f287c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f287c1121c0 con 0x7f287c198620 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f287c112040 con 0x7f287c107280 2024-04-23T18:21:47.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f2882b37640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f287c112340 con 0x7f287c102240 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f28810d4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f28810d4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54832/0 (socket says 172.21.15.26:54832) 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f28810d4640 1 -- 172.21.15.26:0/1463631958 learned_addr learned my addr 172.21.15.26:0/1463631958 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f287bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.738+0000 7f28808d3640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 0x7f287c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 -- 172.21.15.26:0/1463631958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 msgr2=0x7f287c19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 0x7f287c19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 -- 172.21.15.26:0/1463631958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 msgr2=0x7f287c19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 -- 172.21.15.26:0/1463631958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287c199140 con 0x7f287c198620 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f287bfff640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f28810d4640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f287400e970 tx=0x7f287400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287400ccc0 con 0x7f287c198620 2024-04-23T18:21:47.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287c199430 con 0x7f287c198620 2024-04-23T18:21:47.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287400ce60 con 0x7f287c198620 2024-04-23T18:21:47.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874010430 con 0x7f287c198620 2024-04-23T18:21:47.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.739+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f287c1a5680 con 0x7f287c198620 2024-04-23T18:21:47.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.740+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f287c0691c0 con 0x7f287c198620 2024-04-23T18:21:47.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.742+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2874004ff0 con 0x7f287c198620 2024-04-23T18:21:47.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.743+0000 7f2879ffb640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 0x7f28640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:47.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.743+0000 7f28808d3640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 0x7f28640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:47.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.743+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f28740d0b60 con 0x7f287c198620 2024-04-23T18:21:47.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.743+0000 7f28808d3640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 0x7f28640a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f286c005c40 tx=0x7f286c00a570 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:47.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:47.747+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f287409d230 con 0x7f287c198620 2024-04-23T18:21:47.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.005+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f287c10a4a0 con 0x7f287c198620 2024-04-23T18:21:47.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.006+0000 7f2879ffb640 1 -- 172.21.15.26:0/1463631958 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11887 (secure 0 0 0) 0x7f287409cb30 con 0x7f287c198620 2024-04-23T18:21:47.999 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:21:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:21:40.670802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:21:48.000 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2024-04-23T18:21:48.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 msgr2=0x7f28640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 0x7f28640a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f286c005c40 tx=0x7f286c00a570 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 msgr2=0x7f287c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f287400e970 tx=0x7f287400ed70 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 shutdown_connections 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28640a3d20 0x7f28640a61e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287c107280 0x7f287c19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f287c198620 0x7f287c198ad0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 --2- 172.21.15.26:0/1463631958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f287c102240 0x7f287c19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 >> 172.21.15.26:0/1463631958 conn(0x7f287c0790e0 msgr2=0x7f287c100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 shutdown_connections 2024-04-23T18:21:48.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.008+0000 7f2882b37640 1 -- 172.21.15.26:0/1463631958 wait complete. 2024-04-23T18:21:48.013 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 81.1 --await-for 42.0' 2024-04-23T18:21:48.013 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 81.1 --await-for 42.0 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 -- 172.21.15.26:0/1165956492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1010f0 msgr2=0x7f42fc1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 --2- 172.21.15.26:0/1165956492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1010f0 0x7f42fc1014d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009860 tx=0x7f42f802a340 comp rx=0 tx=0).stop 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 -- 172.21.15.26:0/1165956492 shutdown_connections 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 --2- 172.21.15.26:0/1165956492 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc10dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 --2- 172.21.15.26:0/1165956492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc101aa0 0x7f42fc105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 --2- 172.21.15.26:0/1165956492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1010f0 0x7f42fc1014d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 -- 172.21.15.26:0/1165956492 >> 172.21.15.26:0/1165956492 conn(0x7f42fc076f60 msgr2=0x7f42fc077370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 -- 172.21.15.26:0/1165956492 shutdown_connections 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.142+0000 7f43034a0640 1 -- 172.21.15.26:0/1165956492 wait complete. 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 Processor -- start 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 -- start start 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 0x7f42fc19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1985b0 0x7f42fc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1110a0 con 0x7f42fc106140 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc110f20 con 0x7f42fc1985b0 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f43034a0640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc111220 con 0x7f42fc1010f0 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f430123c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 0x7f42fc19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49284/0 (socket says 172.21.15.26:49284) 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f430123c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 0x7f42fc19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:33024/0 (socket says 172.21.15.26:33024) 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 -- 172.21.15.26:0/2629757316 learned_addr learned my addr 172.21.15.26:0/2629757316 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4301a3d640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1985b0 0x7f42fc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 -- 172.21.15.26:0/2629757316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 msgr2=0x7f42fc19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 0x7f42fc19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 -- 172.21.15.26:0/2629757316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1985b0 msgr2=0x7f42fc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1985b0 0x7f42fc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.143+0000 7f4300a3b640 1 -- 172.21.15.26:0/2629757316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc1990d0 con 0x7f42fc106140 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00e9f0 tx=0x7f42ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f42f27fc640 1 -- 172.21.15.26:0/2629757316 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00ccc0 con 0x7f42fc106140 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc1993c0 con 0x7f42fc106140 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f42f27fc640 1 -- 172.21.15.26:0/2629757316 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ec0054d0 con 0x7f42fc106140 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f42f27fc640 1 -- 172.21.15.26:0/2629757316 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec010430 con 0x7f42fc106140 2024-04-23T18:21:48.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.144+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc1a55c0 con 0x7f42fc106140 2024-04-23T18:21:48.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.146+0000 7f42f27fc640 1 -- 172.21.15.26:0/2629757316 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f42ec0046c0 con 0x7f42fc106140 2024-04-23T18:21:48.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 Processor -- start 2024-04-23T18:21:48.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 -- start start 2024-04-23T18:21:48.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f42f27fc640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 0x7f42e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 0x55892b0b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0ba4a0 0x55892b0bc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55892b010370 con 0x55892b0ab8c0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55892afad2e0 con 0x55892b0aafa0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f43061b8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55892b00c330 con 0x55892b0ba4a0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f42f27fc640 1 -- 172.21.15.26:0/2629757316 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f42ec0d0b20 con 0x7f42fc106140 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f4301a3d640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 0x7f42e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 0x55892b0b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:49662/0 (socket says 172.21.15.26:49662) 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 -- 172.21.15.26:0/580727910 learned_addr learned my addr 172.21.15.26:0/580727910 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f4301a3d640 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0ba4a0 0x55892b0bc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 -- 172.21.15.26:0/580727910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0ba4a0 msgr2=0x55892b0bc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0ba4a0 0x55892b0bc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 -- 172.21.15.26:0/580727910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 msgr2=0x55892b0b9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 0x55892b0b9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f430123c640 1 -- 172.21.15.26:0/580727910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55892adeed80 con 0x55892b0aafa0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.147+0000 7f4300a3b640 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 0x55892b0b9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.148+0000 7f4301a3d640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 0x7f42e80a61e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55892b0a86d0 tx=0x7f42f40095c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.148+0000 7f430123c640 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f42f8034000 tx=0x7f42f8005b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c39a4b4c7f3b0ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.148+0000 7f42e77fe640 1 -- 172.21.15.26:0/580727910 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8037040 con 0x55892b0aafa0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.148+0000 7f42e77fe640 1 -- 172.21.15.26:0/580727910 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42f8030dd0 con 0x55892b0aafa0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.148+0000 7f42e77fe640 1 -- 172.21.15.26:0/580727910 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f80038c0 con 0x55892b0aafa0 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- 172.21.15.26:0/580727910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 msgr2=0x55892b0ab380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f42f8034000 tx=0x7f42f8005b10 comp rx=0 tx=0).stop 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- 172.21.15.26:0/580727910 shutdown_connections 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0ba4a0 0x55892b0bc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ab8c0 0x55892b0b9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 --2- 172.21.15.26:0/580727910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0aafa0 0x55892b0ab380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- 172.21.15.26:0/580727910 >> 172.21.15.26:0/580727910 conn(0x55892b0a73c0 msgr2=0x55892b0a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- 172.21.15.26:0/580727910 shutdown_connections 2024-04-23T18:21:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- 172.21.15.26:0/580727910 wait complete. 2024-04-23T18:21:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 -- start start 2024-04-23T18:21:48.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.149+0000 7f43061b8740 1 Processor -- start 2024-04-23T18:21:48.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55892b0bd200 con 0x55892b0ba4a0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55892afaafb0 con 0x55892b0ab8c0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55892b0bd380 con 0x55892b0aafa0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f43061b8740 10 client.0 _ll_register_callbacks cb 0x55892afaa9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f430123c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f430123c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:33040/0 (socket says 172.21.15.26:33040) 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f430123c640 1 -- 172.21.15.26:0/2948213202 learned_addr learned my addr 172.21.15.26:0/2948213202 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4301a3d640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 -- 172.21.15.26:0/2948213202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 msgr2=0x55892b0b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 -- 172.21.15.26:0/2948213202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 msgr2=0x55892b0b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0b7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 -- 172.21.15.26:0/2948213202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55892ae4f510 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f4300a3b640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b7920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0ce2a0 tx=0x7f42ec09c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64c936fd994f4d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.150+0000 7f42e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec005680 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f42ec099160 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42f8030dd0 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42ec09d1f0 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 10 client.25640 fetch_fsmap learned FSMap version 33 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55892ace2670 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42f8030dd0 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f42ec0a52d0 con 0x55892b0ab8c0 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 10 client.25640 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 10 client.25640 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:21:48.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 20 client.25640 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b0ca7d0 0x55892b0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 4 client.25640 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.151+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55892b01dd90 con 0x55892b0ca7d0 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.152+0000 7f430123c640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b0ca7d0 0x55892b0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.152+0000 7f430123c640 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b0ca7d0 0x55892b0ccbf0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b48ecae34321395 server_cookie=494d0aa99cbb2348 in_seq=0 out_seq=0 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.152+0000 7f42e4ff9640 10 client.25640 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.152+0000 7f42e4ff9640 1 -- 172.21.15.26:0/2948213202 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55892b01dd90 con 0x55892b0ca7d0 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f42e4ff9640 10 client.25640 handle_command_reply: tid=0 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 client.25640 shutdown 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 msgr2=0x55892b0b7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b7920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0ce2a0 tx=0x7f42ec09c540 comp rx=0 tx=0).stop 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 shutdown_connections 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b0ca7d0 0x55892b0ccbf0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b7920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f42e4ff9640 0 client.25640 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 --2- 172.21.15.26:0/2948213202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 >> 172.21.15.26:0/2948213202 conn(0x55892b0a73c0 msgr2=0x55892b0ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 shutdown_connections 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 1 -- 172.21.15.26:0/2948213202 wait complete. 2024-04-23T18:21:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.153+0000 7f43061b8740 20 client.25640 trim_cache size 0 max 16384 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 Processor -- start 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 -- start start 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1ae210 con 0x55892b0ba4a0 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1aba80 con 0x55892b0ab8c0 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f43061b8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1a98f0 con 0x55892b0aafa0 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f4300a3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:49674/0 (socket says 172.21.15.26:49674) 2024-04-23T18:21:48.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f4300a3b640 1 -- 172.21.15.26:0/804102254 learned_addr learned my addr 172.21.15.26:0/804102254 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f430123c640 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.172+0000 7f4301a3d640 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 -- 172.21.15.26:0/804102254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 msgr2=0x55892b0c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 -- 172.21.15.26:0/804102254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 msgr2=0x55892b0d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 -- 172.21.15.26:0/804102254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55892b0c9350 con 0x55892b0ab8c0 2024-04-23T18:21:48.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f4300a3b640 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42ec09ca80 tx=0x7f42ec09b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95c153a337ee46 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f42e6ffd640 1 -- 172.21.15.26:0/804102254 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec005680 con 0x55892b0ab8c0 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f42e6ffd640 1 -- 172.21.15.26:0/804102254 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ec09caf0 con 0x55892b0ab8c0 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.173+0000 7f42e6ffd640 1 -- 172.21.15.26:0/804102254 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec098910 con 0x55892b0ab8c0 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- 172.21.15.26:0/804102254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 msgr2=0x55892b0a6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42ec09ca80 tx=0x7f42ec09b000 comp rx=0 tx=0).stop 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- 172.21.15.26:0/804102254 shutdown_connections 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0a6d80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b0d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 --2- 172.21.15.26:0/804102254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- 172.21.15.26:0/804102254 >> 172.21.15.26:0/804102254 conn(0x55892b0a73c0 msgr2=0x55892b0b5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- 172.21.15.26:0/804102254 shutdown_connections 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- 172.21.15.26:0/804102254 wait complete. 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 -- start start 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.174+0000 7f43061b8740 1 Processor -- start 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1a9f40 con 0x55892b0ba4a0 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1ab3b0 con 0x55892b0ab8c0 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc1aa610 con 0x55892b0aafa0 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f43061b8740 10 client.0 _ll_register_callbacks cb 0x7f42ec09c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:48.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49326/0 (socket says 172.21.15.26:49326) 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 -- 172.21.15.26:0/4200085292 learned_addr learned my addr 172.21.15.26:0/4200085292 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f430123c640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4300a3b640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 -- 172.21.15.26:0/4200085292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 msgr2=0x55892b0b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 -- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 msgr2=0x55892b0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 -- 172.21.15.26:0/4200085292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55892b130270 con 0x55892b0ba4a0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f4301a3d640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f42f400ef50 tx=0x7f42f4010040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d88ad8eba63845 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f42e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.175+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4017040 con 0x55892b0ba4a0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f42f4020450 con 0x55892b0ba4a0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42ec09caf0 con 0x55892b0ba4a0 2024-04-23T18:21:48.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f42f4037050 con 0x55892b0ba4a0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 10 client.16191 fetch_fsmap learned FSMap version 33 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55892b1314c0 con 0x55892b0ba4a0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ec09caf0 con 0x55892b0ba4a0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f42f400b040 con 0x55892b0ba4a0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 10 client.16191 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 10 client.16191 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 20 client.16191 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b131dc0 0x55892b1341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 4 client.16191 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await_for": 42.0}] 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await_for": 42.0}) -- 0x55892b017c00 con 0x55892b131dc0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.176+0000 7f430123c640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b131dc0 0x55892b1341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.177+0000 7f430123c640 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b131dc0 0x55892b1341e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0552b8bedeab111 server_cookie=8d74bc7d2034cfc6 in_seq=0 out_seq=0 2024-04-23T18:21:48.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.177+0000 7f42e4ff9640 10 client.16191 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f42e4ff9640 1 -- 172.21.15.26:0/4200085292 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55892b017c00 con 0x55892b131dc0 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f42e4ff9640 10 client.16191 handle_command_reply: tid=0 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 client.16191 shutdown 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 msgr2=0x55892b12fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f42f400ef50 tx=0x7f42f4010040 comp rx=0 tx=0).stop 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 shutdown_connections 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55892b131dc0 0x55892b1341e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55892b0ab8c0 0x55892b0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55892b0ba4a0 0x55892b12fd30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 --2- 172.21.15.26:0/4200085292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55892b0aafa0 0x55892b0b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f42e4ff9640 0 client.16191 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.258+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 >> 172.21.15.26:0/4200085292 conn(0x55892b0a73c0 msgr2=0x55892b0b73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 shutdown_connections 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43061b8740 1 -- 172.21.15.26:0/4200085292 wait complete. 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43061b8740 20 client.16191 trim_cache size 0 max 16384 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 33, 2024-04-23T18:21:48.253 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 22, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "6b33d2ff": { 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 22, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 81.1, 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:21:48.254 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:21:48.255 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 msgr2=0x7f42e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 0x7f42e80a61e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55892b0a86d0 tx=0x7f42f40095c0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 msgr2=0x7f42fc19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00e9f0 tx=0x7f42ec00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 shutdown_connections 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f42e80a3d20 0x7f42e80a61e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42fc1985b0 0x7f42fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f42fc106140 0x7f42fc19e940 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 --2- 172.21.15.26:0/2629757316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc1010f0 0x7f42fc19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 >> 172.21.15.26:0/2629757316 conn(0x7f42fc076f60 msgr2=0x7f42fc06a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 shutdown_connections 2024-04-23T18:21:48.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:21:48.259+0000 7f43034a0640 1 -- 172.21.15.26:0/2629757316 wait complete. 2024-04-23T18:21:48.263 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6b33d2ff', quiesce took 0.6 seconds. Will release after: 21.1 2024-04-23T18:21:48.669 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 8.17 MB/sec execute 103 sec latency 448.127 ms 2024-04-23T18:21:49.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:48 smithi110 ceph-mon[32585]: pgmap v143: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:49.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:48 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1463631958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:49.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:48 smithi026 ceph-mon[29364]: pgmap v143: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:49.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:48 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1463631958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:49.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:48 smithi131 ceph-mon[33041]: pgmap v143: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:49.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:48 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1463631958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 -- 172.21.15.26:0/2907844537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c106f30 msgr2=0x7fd44c107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 --2- 172.21.15.26:0/2907844537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c106f30 0x7fd44c107310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd448009930 tx=0x7fd44802a420 comp rx=0 tx=0).stop 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 -- 172.21.15.26:0/2907844537 shutdown_connections 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 --2- 172.21.15.26:0/2907844537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100450 0x7fd44c1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 --2- 172.21.15.26:0/2907844537 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c100e10 0x7fd44c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 --2- 172.21.15.26:0/2907844537 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c106f30 0x7fd44c107310 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 -- 172.21.15.26:0/2907844537 >> 172.21.15.26:0/2907844537 conn(0x7fd44c0773d0 msgr2=0x7fd44c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 -- 172.21.15.26:0/2907844537 shutdown_connections 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.392+0000 7fd4537bb640 1 -- 172.21.15.26:0/2907844537 wait complete. 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 Processor -- start 2024-04-23T18:21:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 -- start start 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100e10 0x7fd44c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c106f30 0x7fd44c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1122f0 con 0x7fd44c100450 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c112170 con 0x7fd44c100e10 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd4537bb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c112470 con 0x7fd44c106f30 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd451557640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd451557640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49342/0 (socket says 172.21.15.26:49342) 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.393+0000 7fd451557640 1 -- 172.21.15.26:0/13177509 learned_addr learned my addr 172.21.15.26:0/13177509 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd450d56640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100e10 0x7fd44c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451d58640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c106f30 0x7fd44c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 -- 172.21.15.26:0/13177509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c106f30 msgr2=0x7fd44c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c106f30 0x7fd44c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 -- 172.21.15.26:0/13177509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100e10 msgr2=0x7fd44c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100e10 0x7fd44c19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 -- 172.21.15.26:0/13177509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c198d50 con 0x7fd44c100450 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd451557640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd448004770 tx=0x7fd4480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd4427fc640 1 -- 172.21.15.26:0/13177509 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44803f070 con 0x7fd44c100450 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd4427fc640 1 -- 172.21.15.26:0/13177509 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd44803b460 con 0x7fd44c100450 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c198fe0 con 0x7fd44c100450 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd4427fc640 1 -- 172.21.15.26:0/13177509 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448035070 con 0x7fd44c100450 2024-04-23T18:21:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.394+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c1a52e0 con 0x7fd44c100450 2024-04-23T18:21:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.396+0000 7fd4427fc640 1 -- 172.21.15.26:0/13177509 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd44803b8d0 con 0x7fd44c100450 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 Processor -- start 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 -- start start 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ead900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996ebf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55c996e12970 con 0x55c996eade40 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c996daf4a0 con 0x55c996ebcc10 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4565af740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55c996e0e930 con 0x55c996ead520 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd4427fc640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 0x7fd4380a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd451557640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd451557640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49348/0 (socket says 172.21.15.26:49348) 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd451557640 1 -- 172.21.15.26:0/1021189787 learned_addr learned my addr 172.21.15.26:0/1021189787 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd451d58640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 0x7fd4380a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.397+0000 7fd450d56640 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996ebf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451d58640 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ead900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd4427fc640 1 -- 172.21.15.26:0/13177509 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd4480f0b00 con 0x7fd44c100450 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 -- 172.21.15.26:0/1021189787 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 msgr2=0x55c996ead900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ead900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 -- 172.21.15.26:0/1021189787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 msgr2=0x55c996ebf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996ebf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 -- 172.21.15.26:0/1021189787 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c996b25210 con 0x55c996eade40 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451557640 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd448009900 tx=0x7fd4480b8e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3aeef26f4f4dbc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd451d58640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 0x7fd4380a6300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4440148c0 tx=0x7fd4440068e0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd4377fe640 1 -- 172.21.15.26:0/1021189787 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448048020 con 0x55c996eade40 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd4377fe640 1 -- 172.21.15.26:0/1021189787 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4480bd170 con 0x55c996eade40 2024-04-23T18:21:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.398+0000 7fd4377fe640 1 -- 172.21.15.26:0/1021189787 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4480ee4a0 con 0x55c996eade40 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- 172.21.15.26:0/1021189787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 msgr2=0x55c996ebc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd448009900 tx=0x7fd4480b8e40 comp rx=0 tx=0).stop 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- 172.21.15.26:0/1021189787 shutdown_connections 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996ebf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ebc6d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 --2- 172.21.15.26:0/1021189787 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ead900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- 172.21.15.26:0/1021189787 >> 172.21.15.26:0/1021189787 conn(0x55c996ea9b30 msgr2=0x55c996ea9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- 172.21.15.26:0/1021189787 shutdown_connections 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- 172.21.15.26:0/1021189787 wait complete. 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 -- start start 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.399+0000 7fd4565af740 1 Processor -- start 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ebc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 0x55c996eb9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55c996ebf600 con 0x55c996ebcc10 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c996dad0b0 con 0x55c996eade40 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55c996ebf780 con 0x55c996ead520 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd4565af740 10 client.0 _ll_register_callbacks cb 0x55c996d9ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49358/0 (socket says 172.21.15.26:49358) 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 -- 172.21.15.26:0/2147457498 learned_addr learned my addr 172.21.15.26:0/2147457498 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451557640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ebc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd450d56640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 0x55c996eb9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 -- 172.21.15.26:0/2147457498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 msgr2=0x55c996ebc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ebc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 -- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 msgr2=0x55c996eb9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 0x55c996eb9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd451d58640 1 -- 172.21.15.26:0/2147457498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c996bf1620 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.400+0000 7fd450d56640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 0x55c996eb9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd451d58640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd444018660 tx=0x7fd444005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a7c425c3e194bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444027060 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd444022620 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4480bd170 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd44401e040 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd4565af740 10 client.16203 fetch_fsmap learned FSMap version 33 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c996ecb4c0 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4480bd170 con 0x55c996ebcc10 2024-04-23T18:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.401+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fd4440215c0 con 0x55c996ebcc10 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 10 client.16203 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 20 client.16203 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996ecc9f0 0x55c996ecee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 4 client.16203 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c996e20390 con 0x55c996ecc9f0 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd451557640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996ecc9f0 0x55c996ecee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd451557640 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996ecc9f0 0x55c996ecee20 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b6bdd55ff73576a server_cookie=315828366010392b in_seq=0 out_seq=0 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.402+0000 7fd434ff9640 10 client.16203 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd434ff9640 1 -- 172.21.15.26:0/2147457498 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c996e20390 con 0x55c996ecc9f0 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd434ff9640 10 client.16203 handle_command_reply: tid=0 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 client.16203 shutdown 2024-04-23T18:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 msgr2=0x55c996eb9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd444018660 tx=0x7fd444005df0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 shutdown_connections 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996ecc9f0 0x55c996ecee20 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996eade40 0x55c996eb9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996ebcc10 0x55c996eb9e10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd434ff9640 0 client.16203 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 --2- 172.21.15.26:0/2147457498 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996ebc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.403+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 >> 172.21.15.26:0/2147457498 conn(0x55c996ea9b30 msgr2=0x55c996eaf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.404+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 shutdown_connections 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.404+0000 7fd4565af740 1 -- 172.21.15.26:0/2147457498 wait complete. 2024-04-23T18:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.404+0000 7fd4565af740 20 client.16203 trim_cache size 0 max 16384 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.420+0000 7fd4565af740 1 Processor -- start 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.420+0000 7fd4565af740 1 -- start start 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.420+0000 7fd4565af740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ead520 0x55c996ec8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd4565af740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd4565af740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ebcc10 0x55c996ebc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd4565af740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ae170 con 0x55c996eade40 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd4565af740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ab9e0 con 0x55c996ead520 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd4565af740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1a9850 con 0x55c996ebcc10 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49364/0 (socket says 172.21.15.26:49364) 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 -- 172.21.15.26:0/354805711 learned_addr learned my addr 172.21.15.26:0/354805711 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd451557640 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ead520 0x55c996ec8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd451d58640 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ebcc10 0x55c996ebc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 -- 172.21.15.26:0/354805711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ebcc10 msgr2=0x55c996ebc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ebcc10 0x55c996ebc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 -- 172.21.15.26:0/354805711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ead520 msgr2=0x55c996ec8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ead520 0x55c996ec8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 -- 172.21.15.26:0/354805711 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c996eadc70 con 0x55c996eade40 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd450d56640 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005d00 tx=0x7fd43c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f29eca9565d9f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd436ffd640 1 -- 172.21.15.26:0/354805711 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c01f070 con 0x55c996eade40 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd436ffd640 1 -- 172.21.15.26:0/354805711 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd43c005020 con 0x55c996eade40 2024-04-23T18:21:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.421+0000 7fd436ffd640 1 -- 172.21.15.26:0/354805711 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c004910 con 0x55c996eade40 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- 172.21.15.26:0/354805711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 msgr2=0x55c996ea8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005d00 tx=0x7fd43c012820 comp rx=0 tx=0).stop 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- 172.21.15.26:0/354805711 shutdown_connections 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ebcc10 0x55c996ebc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996ea8b30 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 --2- 172.21.15.26:0/354805711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ead520 0x55c996ec8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- 172.21.15.26:0/354805711 >> 172.21.15.26:0/354805711 conn(0x55c996ea9b30 msgr2=0x55c996ea8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- 172.21.15.26:0/354805711 shutdown_connections 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- 172.21.15.26:0/354805711 wait complete. 2024-04-23T18:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.422+0000 7fd4565af740 1 -- start start 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 Processor -- start 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996eb6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996eb6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1a9ea0 con 0x55c996eade40 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ab310 con 0x55c996ebcc10 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1aa570 con 0x55c996ead520 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd4565af740 10 client.0 _ll_register_callbacks cb 0x7fd444016e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd450d56640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd450d56640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49368/0 (socket says 172.21.15.26:49368) 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd450d56640 1 -- 172.21.15.26:0/4291355956 learned_addr learned my addr 172.21.15.26:0/4291355956 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd451d58640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996eb6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd451557640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996eb6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.423+0000 7fd450d56640 1 -- 172.21.15.26:0/4291355956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 msgr2=0x55c996eb6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd450d56640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996eb6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd450d56640 1 -- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 msgr2=0x55c996eb6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd450d56640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996eb6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd450d56640 1 -- 172.21.15.26:0/4291355956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c996f31ff0 con 0x55c996eade40 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd451557640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996eb6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd450d56640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005460 tx=0x7fd43c005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b7ae470f481512 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c01f040 con 0x55c996eade40 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd43c004020 con 0x55c996eade40 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd43c005020 con 0x55c996eade40 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd43c004220 con 0x55c996eade40 2024-04-23T18:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 10 client.16209 fetch_fsmap learned FSMap version 33 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c996f33280 con 0x55c996eade40 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd43c005020 con 0x55c996eade40 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fd43c01a070 con 0x55c996eade40 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 10 client.16209 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 10 client.16209 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.424+0000 7fd4565af740 20 client.16209 populate_metadata read hostname 'smithi026' 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996f33a30 0x55c996f35e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd4565af740 4 client.16209 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c996e1a200 con 0x55c996f33a30 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd451557640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996f33a30 0x55c996f35e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd451557640 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996f33a30 0x55c996f35e50 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6334b0d6a5369b server_cookie=3b4cde0979da9d9 in_seq=0 out_seq=0 2024-04-23T18:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.425+0000 7fd434ff9640 10 client.16209 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd434ff9640 1 -- 172.21.15.26:0/4291355956 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c996e1a200 con 0x55c996f33a30 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd434ff9640 10 client.16209 handle_command_reply: tid=0 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 client.16209 shutdown 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 msgr2=0x55c996eb6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005460 tx=0x7fd43c005490 comp rx=0 tx=0).stop 2024-04-23T18:21:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 shutdown_connections 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55c996f33a30 0x55c996f35e50 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c996ebcc10 0x55c996eb6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55c996eade40 0x55c996eb6aa0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd434ff9640 0 client.16209 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 --2- 172.21.15.26:0/4291355956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c996ead520 0x55c996eb6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 >> 172.21.15.26:0/4291355956 conn(0x55c996ea9b30 msgr2=0x55c996eb4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.463+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 shutdown_connections 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4565af740 1 -- 172.21.15.26:0/4291355956 wait complete. 2024-04-23T18:21:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4565af740 20 client.16209 trim_cache size 0 max 16384 2024-04-23T18:21:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:21:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:21:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:21:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:21:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 msgr2=0x7fd4380a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 0x7fd4380a6300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4440148c0 tx=0x7fd4440068e0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 msgr2=0x7fd44c19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd448004770 tx=0x7fd4480042f0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 shutdown_connections 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd4380a3e40 0x7fd4380a6300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44c100e10 0x7fd44c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd44c106f30 0x7fd44c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 --2- 172.21.15.26:0/13177509 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd44c100450 0x7fd44c19e420 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 >> 172.21.15.26:0/13177509 conn(0x7fd44c0773d0 msgr2=0x7fd44c104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 shutdown_connections 2024-04-23T18:21:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:21:49.464+0000 7fd4537bb640 1 -- 172.21.15.26:0/13177509 wait complete. 2024-04-23T18:21:49.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 8.09 MB/sec execute 104 sec latency 1448.217 ms 2024-04-23T18:21:50.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 8.01 MB/sec execute 105 sec latency 2448.310 ms 2024-04-23T18:21:51.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:50 smithi110 ceph-mon[32585]: pgmap v144: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:50 smithi026 ceph-mon[29364]: pgmap v144: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:51.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:50 smithi131 ceph-mon[33041]: pgmap v144: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 273 op/s 2024-04-23T18:21:51.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.94 MB/sec execute 106 sec latency 3448.430 ms 2024-04-23T18:21:52.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.86 MB/sec execute 107 sec latency 4448.490 ms 2024-04-23T18:21:53.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:52 smithi110 ceph-mon[32585]: pgmap v145: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 384 op/s 2024-04-23T18:21:53.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:52 smithi026 ceph-mon[29364]: pgmap v145: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 384 op/s 2024-04-23T18:21:53.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:52 smithi131 ceph-mon[33041]: pgmap v145: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 384 op/s 2024-04-23T18:21:53.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.79 MB/sec execute 108 sec latency 5448.580 ms 2024-04-23T18:21:54.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.72 MB/sec execute 109 sec latency 6448.698 ms 2024-04-23T18:21:55.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:54 smithi110 ceph-mon[32585]: pgmap v146: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-04-23T18:21:55.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:54 smithi026 ceph-mon[29364]: pgmap v146: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-04-23T18:21:55.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:54 smithi131 ceph-mon[33041]: pgmap v146: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-04-23T18:21:55.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.65 MB/sec execute 110 sec latency 7448.762 ms 2024-04-23T18:21:56.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.58 MB/sec execute 111 sec latency 8448.869 ms 2024-04-23T18:21:57.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:56 smithi110 ceph-mon[32585]: pgmap v147: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 261 op/s 2024-04-23T18:21:57.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:56 smithi026 ceph-mon[29364]: pgmap v147: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 261 op/s 2024-04-23T18:21:57.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:56 smithi131 ceph-mon[33041]: pgmap v147: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 261 op/s 2024-04-23T18:21:57.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.51 MB/sec execute 112 sec latency 9448.953 ms 2024-04-23T18:21:58.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.45 MB/sec execute 113 sec latency 10449.062 ms 2024-04-23T18:21:59.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:21:58 smithi110 ceph-mon[32585]: pgmap v148: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:21:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:21:58 smithi026 ceph-mon[29364]: pgmap v148: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:21:59.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:21:58 smithi131 ceph-mon[33041]: pgmap v148: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:21:59.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.38 MB/sec execute 114 sec latency 11449.174 ms 2024-04-23T18:22:00.670 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.32 MB/sec execute 115 sec latency 12449.266 ms 2024-04-23T18:22:01.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:00 smithi110 ceph-mon[32585]: pgmap v149: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:22:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:00 smithi026 ceph-mon[29364]: pgmap v149: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:22:01.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:00 smithi131 ceph-mon[33041]: pgmap v149: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-04-23T18:22:01.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.25 MB/sec execute 116 sec latency 13449.408 ms 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 -- 172.21.15.26:0/3770347146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 msgr2=0x7fc98410e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 --2- 172.21.15.26:0/3770347146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc98410e3f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc978009900 tx=0x7fc97802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 -- 172.21.15.26:0/3770347146 shutdown_connections 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 --2- 172.21.15.26:0/3770347146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 0x7fc984110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 --2- 172.21.15.26:0/3770347146 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc98410e3f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 --2- 172.21.15.26:0/3770347146 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 0x7fc984101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 -- 172.21.15.26:0/3770347146 >> 172.21.15.26:0/3770347146 conn(0x7fc9840771d0 msgr2=0x7fc9840775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 -- 172.21.15.26:0/3770347146 shutdown_connections 2024-04-23T18:22:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.562+0000 7fc989213640 1 -- 172.21.15.26:0/3770347146 wait complete. 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.563+0000 7fc989213640 1 Processor -- start 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.563+0000 7fc989213640 1 -- start start 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 0x7fc9841a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 0x7fc98419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc984114430 con 0x7fc9841016a0 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841142b0 con 0x7fc98410e930 2024-04-23T18:22:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc989213640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841145b0 con 0x7fc984100d80 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54530/0 (socket says 172.21.15.26:54530) 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 -- 172.21.15.26:0/4189244170 learned_addr learned my addr 172.21.15.26:0/4189244170 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc982d9d640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 0x7fc9841a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98359e640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 0x7fc98419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 -- 172.21.15.26:0/4189244170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 msgr2=0x7fc9841a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 0x7fc9841a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 -- 172.21.15.26:0/4189244170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 msgr2=0x7fc98419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 0x7fc98419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 -- 172.21.15.26:0/4189244170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98419af90 con 0x7fc9841016a0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.564+0000 7fc98259c640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc978005940 tx=0x7fc978004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.565+0000 7fc96bfff640 1 -- 172.21.15.26:0/4189244170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97803f030 con 0x7fc9841016a0 2024-04-23T18:22:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.565+0000 7fc96bfff640 1 -- 172.21.15.26:0/4189244170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc978003aa0 con 0x7fc9841016a0 2024-04-23T18:22:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.565+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98419b220 con 0x7fc9841016a0 2024-04-23T18:22:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.565+0000 7fc96bfff640 1 -- 172.21.15.26:0/4189244170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978035070 con 0x7fc9841016a0 2024-04-23T18:22:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.565+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9841a73b0 con 0x7fc9841016a0 2024-04-23T18:22:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.567+0000 7fc96bfff640 1 -- 172.21.15.26:0/4189244170 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc978042030 con 0x7fc9841016a0 2024-04-23T18:22:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.567+0000 7fc98c009740 1 Processor -- start 2024-04-23T18:22:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.567+0000 7fc98c009740 1 -- start start 2024-04-23T18:22:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af4aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af57bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc96bfff640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 0x7fc96c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56216aeaf970 con 0x56216af4af40 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56216ae4c4a0 con 0x56216af4a620 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98c009740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56216aeab930 con 0x56216af55800 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98359e640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 0x7fc96c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54532/0 (socket says 172.21.15.26:54532) 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc982d9d640 1 -- 172.21.15.26:0/3436796168 learned_addr learned my addr 172.21.15.26:0/3436796168 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af57bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc96bfff640 1 -- 172.21.15.26:0/4189244170 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc9780f0ec0 con 0x7fc9841016a0 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98359e640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af4aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 -- 172.21.15.26:0/3436796168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 msgr2=0x56216af4aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af4aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 -- 172.21.15.26:0/3436796168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 msgr2=0x56216af552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 -- 172.21.15.26:0/3436796168 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216abc2210 con 0x56216af55800 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc982d9d640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98359e640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 0x7fc96c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009270 tx=0x7fc97c009db0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.568+0000 7fc98259c640 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af57bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc9780ee640 tx=0x7fc978040fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc39d3657fe6da1a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc9697fa640 1 -- 172.21.15.26:0/3436796168 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978049040 con 0x56216af55800 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc9697fa640 1 -- 172.21.15.26:0/3436796168 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9780bd590 con 0x56216af55800 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc9697fa640 1 -- 172.21.15.26:0/3436796168 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9780c5b00 con 0x56216af55800 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 -- 172.21.15.26:0/3436796168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 msgr2=0x56216af57bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af57bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc9780ee640 tx=0x7fc978040fb0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 -- 172.21.15.26:0/3436796168 shutdown_connections 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af57bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 --2- 172.21.15.26:0/3436796168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af4aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 -- 172.21.15.26:0/3436796168 >> 172.21.15.26:0/3436796168 conn(0x56216af46b30 msgr2=0x56216af46f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 -- 172.21.15.26:0/3436796168 shutdown_connections 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.569+0000 7fc98c009740 1 -- 172.21.15.26:0/3436796168 wait complete. 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 -- start start 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 Processor -- start 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4af40 0x56216af524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af52ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56216af581f0 con 0x56216af4a620 2024-04-23T18:22:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56216ae4a0b0 con 0x56216af55800 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56216af58370 con 0x56216af4af40 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98c009740 10 client.0 _ll_register_callbacks cb 0x56216ae37d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54542/0 (socket says 172.21.15.26:54542) 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 -- 172.21.15.26:0/1183388665 learned_addr learned my addr 172.21.15.26:0/1183388665 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98359e640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af52ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc98259c640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4af40 0x56216af524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 -- 172.21.15.26:0/1183388665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4af40 msgr2=0x56216af524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4af40 0x56216af524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 -- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 msgr2=0x56216af52ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af52ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.570+0000 7fc982d9d640 1 -- 172.21.15.26:0/1183388665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56216ac8e620 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc982d9d640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc97400bac0 tx=0x7fc97400bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d06736a70eea4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974003e00 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc974011440 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9780bd590 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc974011640 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc98c009740 10 client.16221 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56216af64150 con 0x56216af4a620 2024-04-23T18:22:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9780bd590 con 0x56216af4a620 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fc97401b8a0 con 0x56216af4a620 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc98c009740 10 client.16221 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.571+0000 7fc98c009740 10 client.16221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98c009740 20 client.16221 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216af656d0 0x56216af67ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98c009740 4 client.16221 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56216aebd390 con 0x56216af656d0 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98259c640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216af656d0 0x56216af67ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc966ff5640 10 client.16221 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.572+0000 7fc98259c640 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216af656d0 0x56216af67ac0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f5a05e1e10cf66 server_cookie=ad399b9c0d4a8df8 in_seq=0 out_seq=0 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc966ff5640 1 -- 172.21.15.26:0/1183388665 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56216aebd390 con 0x56216af656d0 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc966ff5640 10 client.16221 handle_command_reply: tid=0 2024-04-23T18:22:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 client.16221 shutdown 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 msgr2=0x56216af551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc97400bac0 tx=0x7fc97400bec0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 shutdown_connections 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216af656d0 0x56216af67ac0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af52ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc966ff5640 0 client.16221 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4af40 0x56216af524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 --2- 172.21.15.26:0/1183388665 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4a620 0x56216af551d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 >> 172.21.15.26:0/1183388665 conn(0x56216af46b30 msgr2=0x56216af4c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 shutdown_connections 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 1 -- 172.21.15.26:0/1183388665 wait complete. 2024-04-23T18:22:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.573+0000 7fc98c009740 20 client.16221 trim_cache size 0 max 16384 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 Processor -- start 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 -- start start 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4a620 0x56216af60fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af6d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841b25f0 con 0x56216af4af40 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841afe60 con 0x56216af55800 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98c009740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841adcb0 con 0x56216af4a620 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54552/0 (socket says 172.21.15.26:54552) 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.588+0000 7fc98259c640 1 -- 172.21.15.26:0/3804032165 learned_addr learned my addr 172.21.15.26:0/3804032165 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc982d9d640 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4a620 0x56216af60fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98359e640 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af6d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 -- 172.21.15.26:0/3804032165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4a620 msgr2=0x56216af60fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4a620 0x56216af60fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 -- 172.21.15.26:0/3804032165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 msgr2=0x56216af6d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af6d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 -- 172.21.15.26:0/3804032165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216af514b0 con 0x56216af4af40 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98259c640 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc9780bcb90 tx=0x7fc978049bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7febc65c20aa77b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc968ff9640 1 -- 172.21.15.26:0/3804032165 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978049040 con 0x56216af4af40 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc968ff9640 1 -- 172.21.15.26:0/3804032165 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9780c5b00 con 0x56216af4af40 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc968ff9640 1 -- 172.21.15.26:0/3804032165 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9780b9500 con 0x56216af4af40 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98c009740 1 -- 172.21.15.26:0/3804032165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 msgr2=0x56216af45b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.589+0000 7fc98c009740 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc9780bcb90 tx=0x7fc978049bd0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- 172.21.15.26:0/3804032165 shutdown_connections 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af55800 0x56216af6d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af45b30 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- 172.21.15.26:0/3804032165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af4a620 0x56216af60fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- 172.21.15.26:0/3804032165 >> 172.21.15.26:0/3804032165 conn(0x56216af46b30 msgr2=0x56216af45420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- 172.21.15.26:0/3804032165 shutdown_connections 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- 172.21.15.26:0/3804032165 wait complete. 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- start start 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 Processor -- start 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af5af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af5ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841ae320 con 0x56216af4af40 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841af790 con 0x56216af4a620 2024-04-23T18:22:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841ae9f0 con 0x56216af55800 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98c009740 10 client.0 _ll_register_callbacks cb 0x7fc9740044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54562/0 (socket says 172.21.15.26:54562) 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc98259c640 1 -- 172.21.15.26:0/951365597 learned_addr learned my addr 172.21.15.26:0/951365597 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.590+0000 7fc982d9d640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af5af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98359e640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af5ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 -- 172.21.15.26:0/951365597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 msgr2=0x56216af5ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af5ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 -- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 msgr2=0x56216af5af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af5af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 -- 172.21.15.26:0/951365597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56216af52e40 con 0x56216af4af40 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98359e640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af5ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98259c640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc9780c55d0 tx=0x7fc978005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ba78547b7bba9f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978049050 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc97803bcc0 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9780c5b00 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc9780bcc60 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98c009740 10 client.16227 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56216af67e30 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9780c5b00 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.591+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fc978049c40 con 0x56216af4af40 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 10 client.16227 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 10 client.16227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 20 client.16227 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216afcbf50 0x56216afce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 4 client.16227 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56216aeb7200 con 0x56216afcbf50 2024-04-23T18:22:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc982d9d640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216afcbf50 0x56216afce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc982d9d640 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216afcbf50 0x56216afce370 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7431a60afe8c0bf server_cookie=4ca8319e3a0386e6 in_seq=0 out_seq=0 2024-04-23T18:22:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.592+0000 7fc966ff5640 10 client.16227 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc966ff5640 1 -- 172.21.15.26:0/951365597 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56216aeb7200 con 0x56216afcbf50 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc966ff5640 10 client.16227 handle_command_reply: tid=0 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 client.16227 shutdown 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 msgr2=0x56216af5b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc9780c55d0 tx=0x7fc978005bd0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 shutdown_connections 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56216afcbf50 0x56216afce370 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56216af55800 0x56216af5ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc966ff5640 0 client.16227 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56216af4af40 0x56216af5b4d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 --2- 172.21.15.26:0/951365597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56216af4a620 0x56216af5af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 >> 172.21.15.26:0/951365597 conn(0x56216af46b30 msgr2=0x56216af45cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 shutdown_connections 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 1 -- 172.21.15.26:0/951365597 wait complete. 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.613+0000 7fc98c009740 20 client.16227 trim_cache size 0 max 16384 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:22:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 msgr2=0x7fc96c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 0x7fc96c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009270 tx=0x7fc97c009db0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 msgr2=0x7fc9841a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc978005940 tx=0x7fc978004270 comp rx=0 tx=0).stop 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 shutdown_connections 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc96c0a3dc0 0x7fc96c0a6280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98410e930 0x7fc98419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc9841016a0 0x7fc9841a0b50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 --2- 172.21.15.26:0/4189244170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc984100d80 0x7fc9841a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 >> 172.21.15.26:0/4189244170 conn(0x7fc9840771d0 msgr2=0x7fc98410fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 shutdown_connections 2024-04-23T18:22:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:02.614+0000 7fc989213640 1 -- 172.21.15.26:0/4189244170 wait complete. 2024-04-23T18:22:02.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.19 MB/sec execute 117 sec latency 14449.486 ms 2024-04-23T18:22:03.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:02 smithi110 ceph-mon[32585]: pgmap v150: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T18:22:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:02 smithi026 ceph-mon[29364]: pgmap v150: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T18:22:03.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:02 smithi131 ceph-mon[33041]: pgmap v150: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T18:22:03.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.13 MB/sec execute 118 sec latency 15449.603 ms 2024-04-23T18:22:04.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.07 MB/sec execute 119 sec latency 16449.723 ms 2024-04-23T18:22:05.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:04 smithi110 ceph-mon[32585]: pgmap v151: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-04-23T18:22:05.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:04 smithi026 ceph-mon[29364]: pgmap v151: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-04-23T18:22:05.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:04 smithi131 ceph-mon[33041]: pgmap v151: 97 pgs: 97 active+clean; 164 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-04-23T18:22:05.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 7.01 MB/sec execute 120 sec latency 17449.840 ms 2024-04-23T18:22:06.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 6.95 MB/sec execute 121 sec latency 18449.894 ms 2024-04-23T18:22:07.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:06 smithi110 ceph-mon[32585]: pgmap v152: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-04-23T18:22:07.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:06 smithi026 ceph-mon[29364]: pgmap v152: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-04-23T18:22:07.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:06 smithi131 ceph-mon[33041]: pgmap v152: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-04-23T18:22:07.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 6.90 MB/sec execute 122 sec latency 19449.935 ms 2024-04-23T18:22:08.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 6.84 MB/sec execute 123 sec latency 20450.047 ms 2024-04-23T18:22:09.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:08 smithi110 ceph-mon[32585]: pgmap v153: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:09.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:08 smithi026 ceph-mon[29364]: pgmap v153: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:09.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:08 smithi131 ceph-mon[33041]: pgmap v153: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:09.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- 172.21.15.26:0/3594021099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba8101680 msgr2=0x7fcba810e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 --2- 172.21.15.26:0/3594021099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba8101680 0x7fcba810e3d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009900 tx=0x7fcb9802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- 172.21.15.26:0/3594021099 shutdown_connections 2024-04-23T18:22:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 --2- 172.21.15.26:0/3594021099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba810e910 0x7fcba8110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 --2- 172.21.15.26:0/3594021099 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba8101680 0x7fcba810e3d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 --2- 172.21.15.26:0/3594021099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8100d60 0x7fcba8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- 172.21.15.26:0/3594021099 >> 172.21.15.26:0/3594021099 conn(0x7fcba8077f50 msgr2=0x7fcba8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- 172.21.15.26:0/3594021099 shutdown_connections 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- 172.21.15.26:0/3594021099 wait complete. 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 Processor -- start 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.487+0000 7fcbaf387640 1 -- start start 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba8100d60 0x7fcba81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 0x7fcba81a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fcba8114430 con 0x7fcba810e910 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcba81142b0 con 0x7fcba8101680 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbaf387640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fcba81145b0 con 0x7fcba8100d60 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50650/0 (socket says 172.21.15.26:50650) 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 -- 172.21.15.26:0/2044426936 learned_addr learned my addr 172.21.15.26:0/2044426936 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad123640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba8100d60 0x7fcba81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbac922640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 0x7fcba81a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 -- 172.21.15.26:0/2044426936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba8100d60 msgr2=0x7fcba81a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba8100d60 0x7fcba81a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 -- 172.21.15.26:0/2044426936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 msgr2=0x7fcba81a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 0x7fcba81a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 -- 172.21.15.26:0/2044426936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba819afb0 con 0x7fcba810e910 2024-04-23T18:22:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbac922640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 0x7fcba81a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.488+0000 7fcbad924640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e9d0 tx=0x7fcba000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.489+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba000cd80 con 0x7fcba810e910 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.489+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba0005020 con 0x7fcba810e910 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.489+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba819b2a0 con 0x7fcba810e910 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.489+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba00116d0 con 0x7fcba810e910 2024-04-23T18:22:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.489+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba81a75b0 con 0x7fcba810e910 2024-04-23T18:22:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.490+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba80698e0 con 0x7fcba810e910 2024-04-23T18:22:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.492+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcba00051d0 con 0x7fcba810e910 2024-04-23T18:22:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.492+0000 7fcb9e7fc640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 0x7fcb940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.492+0000 7fcbad123640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 0x7fcb940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.492+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fcba00d0d20 con 0x7fcba810e910 2024-04-23T18:22:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.493+0000 7fcbad123640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 0x7fcb940a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005e10 tx=0x7fcba4005da0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.496+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcba0016030 con 0x7fcba810e910 2024-04-23T18:22:09.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 312586 6.78 MB/sec execute 124 sec latency 21450.099 ms 2024-04-23T18:22:09.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.727+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcba8068d70 con 0x7fcba810e910 2024-04-23T18:22:09.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.728+0000 7fcb9e7fc640 1 -- 172.21.15.26:0/2044426936 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11887 (secure 0 0 0) 0x7fcba009d3f0 con 0x7fcba810e910 2024-04-23T18:22:09.723 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:22:09.723 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:21:40.670802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:22:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2024-04-23T18:22:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 msgr2=0x7fcb940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 0x7fcb940a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005e10 tx=0x7fcba4005da0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 msgr2=0x7fcba819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e9d0 tx=0x7fcba000ee90 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 shutdown_connections 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fcb940a3d20 0x7fcb940a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcba8101680 0x7fcba81a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcba810e910 0x7fcba819a6f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 --2- 172.21.15.26:0/2044426936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcba8100d60 0x7fcba81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 >> 172.21.15.26:0/2044426936 conn(0x7fcba8077f50 msgr2=0x7fcba810fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 shutdown_connections 2024-04-23T18:22:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.730+0000 7fcbaf387640 1 -- 172.21.15.26:0/2044426936 wait complete. 2024-04-23T18:22:09.735 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 6b33d2ff --release --await' 2024-04-23T18:22:09.736 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 6b33d2ff --release --await 2024-04-23T18:22:09.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 -- 172.21.15.26:0/933617374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 msgr2=0x7f80e810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 --2- 172.21.15.26:0/933617374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e810dd30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009ca0 tx=0x7f80e002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 -- 172.21.15.26:0/933617374 shutdown_connections 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 --2- 172.21.15.26:0/933617374 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e810dd30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 --2- 172.21.15.26:0/933617374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e8101a40 0x7f80e8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 --2- 172.21.15.26:0/933617374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 0x7f80e8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 -- 172.21.15.26:0/933617374 >> 172.21.15.26:0/933617374 conn(0x7f80e8077f50 msgr2=0x7f80e8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 -- 172.21.15.26:0/933617374 shutdown_connections 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.850+0000 7f80ed435640 1 -- 172.21.15.26:0/933617374 wait complete. 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 Processor -- start 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 -- start start 2024-04-23T18:22:09.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 0x7f80e8199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e819a060 0x7f80e81a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81110d0 con 0x7f80e8106160 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8110f50 con 0x7f80e8101090 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80ed435640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8111250 con 0x7f80e819a060 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50662/0 (socket says 172.21.15.26:50662) 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80e67fc640 1 -- 172.21.15.26:0/2527459820 learned_addr learned my addr 172.21.15.26:0/2527459820 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.851+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 0x7f80e8199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e819a060 0x7f80e81a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 -- 172.21.15.26:0/2527459820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e819a060 msgr2=0x7f80e81a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e819a060 0x7f80e81a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 -- 172.21.15.26:0/2527459820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 msgr2=0x7f80e8199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 0x7f80e8199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 -- 172.21.15.26:0/2527459820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e81a4e10 con 0x7f80e8106160 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c970 tx=0x7f80dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80cffff640 1 -- 172.21.15.26:0/2527459820 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc007bb0 con 0x7f80e8106160 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80cffff640 1 -- 172.21.15.26:0/2527459820 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc007d50 con 0x7f80e8106160 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e81a5100 con 0x7f80e8106160 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80cffff640 1 -- 172.21.15.26:0/2527459820 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc018730 con 0x7f80e8106160 2024-04-23T18:22:09.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.852+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e81a5600 con 0x7f80e8106160 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.854+0000 7f80cffff640 1 -- 172.21.15.26:0/2527459820 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f80dc00f990 con 0x7f80e8106160 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 Processor -- start 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 -- start start 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa49870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 0x557d9aa53f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x557d9a9ae6d0 con 0x557d9aa54480 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557d9a94b1a0 con 0x557d9aa49490 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80f022b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x557d9a9aa690 con 0x557d9aa49db0 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50670/0 (socket says 172.21.15.26:50670) 2024-04-23T18:22:09.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 -- 172.21.15.26:0/2833635309 learned_addr learned my addr 172.21.15.26:0/2833635309 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80cffff640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 0x7f80d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 0x557d9aa53f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa49870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 0x7f80d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80cffff640 1 -- 172.21.15.26:0/2527459820 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f80dc0d0820 con 0x7f80e8106160 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 -- 172.21.15.26:0/2833635309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 msgr2=0x557d9aa53f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 0x557d9aa53f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 -- 172.21.15.26:0/2833635309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 msgr2=0x557d9aa49870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa49870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e67fc640 1 -- 172.21.15.26:0/2833635309 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d9a9555b0 con 0x557d9aa54480 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.855+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 0x557d9aa53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.856+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f80dc09c490 tx=0x7f80dc00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09309ae28788eca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.856+0000 7f80cd7fa640 1 -- 172.21.15.26:0/2833635309 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc09c0f0 con 0x557d9aa54480 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.856+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 0x7f80d00a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009430 tx=0x7f80e00095c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.856+0000 7f80cd7fa640 1 -- 172.21.15.26:0/2833635309 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc09ce90 con 0x557d9aa54480 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.856+0000 7f80cd7fa640 1 -- 172.21.15.26:0/2833635309 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0ce1c0 con 0x557d9aa54480 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- 172.21.15.26:0/2833635309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 msgr2=0x557d9aa56870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f80dc09c490 tx=0x7f80dc00af90 comp rx=0 tx=0).stop 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- 172.21.15.26:0/2833635309 shutdown_connections 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa56870 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49db0 0x557d9aa53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- 172.21.15.26:0/2833635309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- 172.21.15.26:0/2833635309 >> 172.21.15.26:0/2833635309 conn(0x557d9aa457b0 msgr2=0x557d9aa45b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- 172.21.15.26:0/2833635309 shutdown_connections 2024-04-23T18:22:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- 172.21.15.26:0/2833635309 wait complete. 2024-04-23T18:22:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- start start 2024-04-23T18:22:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 Processor -- start 2024-04-23T18:22:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa51180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa51790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x557d9aa57250 con 0x557d9aa49490 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557d9a948b20 con 0x557d9aa49db0 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.857+0000 7f80f022b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x557d9aa573d0 con 0x557d9aa54480 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80f022b740 10 client.0 _ll_register_callbacks cb 0x557d9a948530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50672/0 (socket says 172.21.15.26:50672) 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 -- 172.21.15.26:0/2279968427 learned_addr learned my addr 172.21.15.26:0/2279968427 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa51180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa51790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 -- 172.21.15.26:0/2279968427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 msgr2=0x557d9aa51790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa51790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 -- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 msgr2=0x557d9aa51180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa51180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 -- 172.21.15.26:0/2279968427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d9a6c0920 con 0x557d9aa49490 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa51180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f80d8010820 tx=0x7f80d8010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95da668c139dadb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.858+0000 7f80caff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:09.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.859+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d800fb90 con 0x557d9aa49490 2024-04-23T18:22:09.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.859+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f80d800fd30 con 0x557d9aa49490 2024-04-23T18:22:09.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.859+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80dc09ce90 con 0x557d9aa49490 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f80d8004020 con 0x557d9aa49490 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 10 client.16245 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557d9a78cd30 con 0x557d9aa49490 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc09ce90 con 0x557d9aa49490 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f80d8019410 con 0x557d9aa49490 2024-04-23T18:22:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 10 client.16245 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 10 client.16245 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 20 client.16245 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aa64620 0x557d9aa66ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 4 client.16245 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d9a96c500 con 0x557d9aa64620 2024-04-23T18:22:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.860+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aa64620 0x557d9aa66ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.861+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aa64620 0x557d9aa66ac0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f683a0121cf94a33 server_cookie=2f35b5d22191b86d in_seq=0 out_seq=0 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.861+0000 7f80caff5640 10 client.16245 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.861+0000 7f80caff5640 1 -- 172.21.15.26:0/2279968427 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d9a96c500 con 0x557d9aa64620 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.861+0000 7f80caff5640 10 client.16245 handle_command_reply: tid=0 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 client.16245 shutdown 2024-04-23T18:22:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 msgr2=0x557d9aa53e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f80d8010820 tx=0x7f80d8010ce0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 shutdown_connections 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aa64620 0x557d9aa66ac0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa51180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa51790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80caff5640 0 client.16245 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 --2- 172.21.15.26:0/2279968427 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49490 0x557d9aa53e90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 >> 172.21.15.26:0/2279968427 conn(0x557d9aa457b0 msgr2=0x557d9aa4b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 shutdown_connections 2024-04-23T18:22:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 1 -- 172.21.15.26:0/2279968427 wait complete. 2024-04-23T18:22:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.862+0000 7f80f022b740 20 client.16245 trim_cache size 0 max 16384 2024-04-23T18:22:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 Processor -- start 2024-04-23T18:22:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 -- start start 2024-04-23T18:22:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49490 0x557d9aa4bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81ae200 con 0x557d9aa54480 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81aba70 con 0x557d9aa49db0 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80f022b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81a98e0 con 0x557d9aa49490 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50688/0 (socket says 172.21.15.26:50688) 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e77fe640 1 -- 172.21.15.26:0/1167711980 learned_addr learned my addr 172.21.15.26:0/1167711980 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e67fc640 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49490 0x557d9aa4bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e77fe640 1 -- 172.21.15.26:0/1167711980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49490 msgr2=0x557d9aa4bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.880+0000 7f80e77fe640 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49490 0x557d9aa4bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80e77fe640 1 -- 172.21.15.26:0/1167711980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 msgr2=0x557d9aa539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80e77fe640 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80e77fe640 1 -- 172.21.15.26:0/1167711980 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d9aa51b00 con 0x557d9aa54480 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80e77fe640 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f80e00098f0 tx=0x7f80e003b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5acf338a478db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80ccff9640 1 -- 172.21.15.26:0/1167711980 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0003970 con 0x557d9aa54480 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80ccff9640 1 -- 172.21.15.26:0/1167711980 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e0036040 con 0x557d9aa54480 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.881+0000 7f80ccff9640 1 -- 172.21.15.26:0/1167711980 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e002f660 con 0x557d9aa54480 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- 172.21.15.26:0/1167711980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 msgr2=0x557d9aa6c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f80e00098f0 tx=0x7f80e003b000 comp rx=0 tx=0).stop 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- 172.21.15.26:0/1167711980 shutdown_connections 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49db0 0x557d9aa539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa54480 0x557d9aa6c390 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- 172.21.15.26:0/1167711980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa49490 0x557d9aa4bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- 172.21.15.26:0/1167711980 >> 172.21.15.26:0/1167711980 conn(0x557d9aa457b0 msgr2=0x557d9aa449d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- 172.21.15.26:0/1167711980 shutdown_connections 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- 172.21.15.26:0/1167711980 wait complete. 2024-04-23T18:22:09.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- start start 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 Processor -- start 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa4e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa5ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81a9f30 con 0x557d9aa49db0 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81ab3a0 con 0x557d9aa49490 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81aa600 con 0x557d9aa54480 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80f022b740 10 client.0 _ll_register_callbacks cb 0x7f80d8010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50694/0 (socket says 172.21.15.26:50694) 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80e67fc640 1 -- 172.21.15.26:0/2888941610 learned_addr learned my addr 172.21.15.26:0/2888941610 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.882+0000 7f80e77fe640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa5ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa4e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 -- 172.21.15.26:0/2888941610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 msgr2=0x557d9aa5ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa5ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 -- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 msgr2=0x557d9aa4e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa4e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 -- 172.21.15.26:0/2888941610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d9aa5b2f0 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80e67fc640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f80dc098bf0 tx=0x7f80dc00a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b013df1f14a99c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc09ce90 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f80dc0ce1c0 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80e0036040 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f80dc0ce3c0 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80f022b740 10 client.16251 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557d9aacb540 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e0036040 con 0x557d9aa49db0 2024-04-23T18:22:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f80dc020030 con 0x557d9aa49db0 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80f022b740 10 client.16251 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.883+0000 7f80f022b740 10 client.16251 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80f022b740 20 client.16251 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aacb880 0x557d9aacdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80f022b740 4 client.16251 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6b33d2ff", "release": true, "await": true}] 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6b33d2ff", "release": true, "await": true}) -- 0x557d9a9bc0f0 con 0x557d9aacb880 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aacb880 0x557d9aacdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80e6ffd640 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aacb880 0x557d9aacdca0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54d8af276845358c server_cookie=5831eac1098fffda in_seq=0 out_seq=0 2024-04-23T18:22:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.884+0000 7f80caff5640 10 client.16251 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80caff5640 1 -- 172.21.15.26:0/2888941610 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557d9a9bc0f0 con 0x557d9aacb880 2024-04-23T18:22:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80caff5640 10 client.16251 handle_command_reply: tid=0 2024-04-23T18:22:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 client.16251 shutdown 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 msgr2=0x557d9aa4ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f80dc098bf0 tx=0x7f80dc00a370 comp rx=0 tx=0).stop 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 shutdown_connections 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x557d9aacb880 0x557d9aacdca0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557d9aa54480 0x557d9aa5ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80caff5640 0 client.16251 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x557d9aa49db0 0x557d9aa4ead0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 --2- 172.21.15.26:0/2888941610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d9aa49490 0x557d9aa4e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 >> 172.21.15.26:0/2888941610 conn(0x557d9aa457b0 msgr2=0x557d9aa5f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 shutdown_connections 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 1 -- 172.21.15.26:0/2888941610 wait complete. 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80f022b740 20 client.16251 trim_cache size 0 max 16384 2024-04-23T18:22:09.888 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 33, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 24, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "6b33d2ff": { 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 24, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 81.1, 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:22:09.889 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:09.890 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 msgr2=0x7f80d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 0x7f80d00a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009430 tx=0x7f80e00095c0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 msgr2=0x7f80e8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c970 tx=0x7f80dc00ce30 comp rx=0 tx=0).stop 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 shutdown_connections 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80d00a3d20 0x7f80d00a61e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80e819a060 0x7f80e81a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80e8106160 0x7f80e8199ad0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 --2- 172.21.15.26:0/2527459820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8101090 0x7f80e8199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.889+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 >> 172.21.15.26:0/2527459820 conn(0x7f80e8077f50 msgr2=0x7f80e80fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.890+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 shutdown_connections 2024-04-23T18:22:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:09.890+0000 7f80ed435640 1 -- 172.21.15.26:0/2527459820 wait complete. 2024-04-23T18:22:09.894 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6b33d2ff', seconds elapsed: 22.3 2024-04-23T18:22:09.895 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.1 2024-04-23T18:22:09.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 -- 172.21.15.26:0/1382120699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53641031b0 msgr2=0x7f5364103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 --2- 172.21.15.26:0/1382120699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53641031b0 0x7f5364103590 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5354009900 tx=0x7f535402a480 comp rx=0 tx=0).stop 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 -- 172.21.15.26:0/1382120699 shutdown_connections 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 --2- 172.21.15.26:0/1382120699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f536410c580 0x7f536410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 --2- 172.21.15.26:0/1382120699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 0x7f536410bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 --2- 172.21.15.26:0/1382120699 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f53641031b0 0x7f5364103590 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 -- 172.21.15.26:0/1382120699 >> 172.21.15.26:0/1382120699 conn(0x7f53640771d0 msgr2=0x7f53640775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 -- 172.21.15.26:0/1382120699 shutdown_connections 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 -- 172.21.15.26:0/1382120699 wait complete. 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.006+0000 7f536a3cc640 1 Processor -- start 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 -- start start 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53641031b0 0x7f536419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 0x7f536419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f53641121d0 con 0x7f536410c580 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5364112050 con 0x7f53641031b0 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f536a3cc640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5364112350 con 0x7f5364103ad0 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5363fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53641031b0 0x7f536419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f53637fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 0x7f536419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50712/0 (socket says 172.21.15.26:50712) 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 -- 172.21.15.26:0/601019782 learned_addr learned my addr 172.21.15.26:0/601019782 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 -- 172.21.15.26:0/601019782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 msgr2=0x7f536419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 0x7f536419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 -- 172.21.15.26:0/601019782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53641031b0 msgr2=0x7f536419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53641031b0 0x7f536419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.007+0000 7f5368969640 1 -- 172.21.15.26:0/601019782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5364198d60 con 0x7f536410c580 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f5368969640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f535c00eb40 tx=0x7f535c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c004b20 con 0x7f536410c580 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5364199050 con 0x7f536410c580 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535c004cc0 con 0x7f536410c580 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c010720 con 0x7f536410c580 2024-04-23T18:22:10.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.008+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53641a52f0 con 0x7f536410c580 2024-04-23T18:22:10.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.009+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536406add0 con 0x7f536410c580 2024-04-23T18:22:10.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.010+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f535c005560 con 0x7f536410c580 2024-04-23T18:22:10.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.011+0000 7f53617fa640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 0x7f534c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.011+0000 7f5363fff640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 0x7f534c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.011+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f535c0d0e90 con 0x7f536410c580 2024-04-23T18:22:10.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.011+0000 7f5363fff640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 0x7f534c0a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5354031440 tx=0x7f5354005b50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.014+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f535c09d500 con 0x7f536410c580 2024-04-23T18:22:10.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:09 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2044426936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:10.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:09 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2044426936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:10.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:09 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2044426936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:10.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.270+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53640691c0 con 0x7f536410c580 2024-04-23T18:22:10.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.270+0000 7f53617fa640 1 -- 172.21.15.26:0/601019782 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11887 (secure 0 0 0) 0x7f535c09ce00 con 0x7f536410c580 2024-04-23T18:22:10.265 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:22:10.265 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:21:40.670802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:22:10.265 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 33 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 msgr2=0x7f534c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 0x7f534c0a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5354031440 tx=0x7f5354005b50 comp rx=0 tx=0).stop 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 msgr2=0x7f53641985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f535c00eb40 tx=0x7f535c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 shutdown_connections 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f534c0a3d20 0x7f534c0a61e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f536410c580 0x7f53641985e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5364103ad0 0x7f536419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 --2- 172.21.15.26:0/601019782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53641031b0 0x7f536419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 >> 172.21.15.26:0/601019782 conn(0x7f53640771d0 msgr2=0x7f53640fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 shutdown_connections 2024-04-23T18:22:10.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.273+0000 7f536a3cc640 1 -- 172.21.15.26:0/601019782 wait complete. 2024-04-23T18:22:10.279 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 81.1 --await-for 42.0' 2024-04-23T18:22:10.279 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 81.1 --await-for 42.0 2024-04-23T18:22:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 -- 172.21.15.26:0/1825799299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80107280 msgr2=0x7fbb8010ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 --2- 172.21.15.26:0/1825799299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80107280 0x7fbb8010ee50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009ca0 tx=0x7fbb7802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 -- 172.21.15.26:0/1825799299 shutdown_connections 2024-04-23T18:22:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 --2- 172.21.15.26:0/1825799299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb80102bf0 0x7fbb80106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 --2- 172.21.15.26:0/1825799299 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80107280 0x7fbb8010ee50 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 --2- 172.21.15.26:0/1825799299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80102240 0x7fbb80102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 -- 172.21.15.26:0/1825799299 >> 172.21.15.26:0/1825799299 conn(0x7fbb800790e0 msgr2=0x7fbb800794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 -- 172.21.15.26:0/1825799299 shutdown_connections 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.404+0000 7fbb87f43640 1 -- 172.21.15.26:0/1825799299 wait complete. 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 Processor -- start 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 -- start start 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80107280 0x7fbb80198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 0x7fbb8019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbb80112250 con 0x7fbb80102240 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801120d0 con 0x7fbb8019d000 2024-04-23T18:22:10.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb87f43640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801123d0 con 0x7fbb80107280 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50716/0 (socket says 172.21.15.26:50716) 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb864e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 0x7fbb8019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb854de640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80107280 0x7fbb80198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.405+0000 7fbb85cdf640 1 -- 172.21.15.26:0/108939441 learned_addr learned my addr 172.21.15.26:0/108939441 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 -- 172.21.15.26:0/108939441 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80107280 msgr2=0x7fbb80198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80107280 0x7fbb80198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 -- 172.21.15.26:0/108939441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 msgr2=0x7fbb8019d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 0x7fbb8019d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 -- 172.21.15.26:0/108939441 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8019daf0 con 0x7fbb80102240 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb864e0640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 0x7fbb8019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00e9f0 tx=0x7fbb7c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb76ffd640 1 -- 172.21.15.26:0/108939441 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c00cd80 con 0x7fbb80102240 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8019dde0 con 0x7fbb80102240 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb76ffd640 1 -- 172.21.15.26:0/108939441 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb7c0054d0 con 0x7fbb80102240 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb76ffd640 1 -- 172.21.15.26:0/108939441 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c010690 con 0x7fbb80102240 2024-04-23T18:22:10.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.406+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb801a55d0 con 0x7fbb80102240 2024-04-23T18:22:10.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb76ffd640 1 -- 172.21.15.26:0/108939441 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb7c004e90 con 0x7fbb80102240 2024-04-23T18:22:10.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 Processor -- start 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 -- start start 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a794b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a797480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a23a6ef370 con 0x55a23a78a0a0 2024-04-23T18:22:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a23a68c2e0 con 0x55a23a795090 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.409+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a23a6eb330 con 0x55a23a78a9c0 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb76ffd640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 0x7fbb6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a794b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50732/0 (socket says 172.21.15.26:50732) 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 -- 172.21.15.26:0/4035553990 learned_addr learned my addr 172.21.15.26:0/4035553990 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 0x7fbb6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb854de640 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a797480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb76ffd640 1 -- 172.21.15.26:0/108939441 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fbb7c0d0bd0 con 0x7fbb80102240 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 -- 172.21.15.26:0/4035553990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 msgr2=0x55a23a794b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a794b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 -- 172.21.15.26:0/4035553990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 msgr2=0x55a23a797480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a797480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 -- 172.21.15.26:0/4035553990 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a23a4cdd80 con 0x55a23a78a0a0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a794b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009430 tx=0x7fbb78031800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d66d89c00aa583 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb864e0640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 0x7fbb6c0a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802aeb0 tx=0x7fbb7803b000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb6bfff640 1 -- 172.21.15.26:0/4035553990 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7804f070 con 0x55a23a78a0a0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb6bfff640 1 -- 172.21.15.26:0/4035553990 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb78036040 con 0x55a23a78a0a0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.410+0000 7fbb6bfff640 1 -- 172.21.15.26:0/4035553990 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7802f990 con 0x55a23a78a0a0 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 -- 172.21.15.26:0/4035553990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 msgr2=0x55a23a78a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009430 tx=0x7fbb78031800 comp rx=0 tx=0).stop 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 -- 172.21.15.26:0/4035553990 shutdown_connections 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a797480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a794b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/4035553990 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a78a480 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 -- 172.21.15.26:0/4035553990 >> 172.21.15.26:0/4035553990 conn(0x55a23a7863c0 msgr2=0x55a23a7867a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 -- 172.21.15.26:0/4035553990 shutdown_connections 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.411+0000 7fbb8ac56740 1 -- 172.21.15.26:0/4035553990 wait complete. 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 -- start start 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 Processor -- start 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 0x55a23a7920c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a795090 0x55a23a792b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a23a797ca0 con 0x55a23a78a9c0 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a23a689fb0 con 0x55a23a78a0a0 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a23a797e20 con 0x55a23a795090 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb8ac56740 10 client.0 _ll_register_callbacks cb 0x55a23a6899a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb854de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 0x55a23a7920c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb854de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50746/0 (socket says 172.21.15.26:50746) 2024-04-23T18:22:10.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb854de640 1 -- 172.21.15.26:0/3302278712 learned_addr learned my addr 172.21.15.26:0/3302278712 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.412+0000 7fbb864e0640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a795090 0x55a23a792b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 -- 172.21.15.26:0/3302278712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a795090 msgr2=0x55a23a792b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a795090 0x55a23a792b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 -- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 msgr2=0x55a23a7920c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 0x55a23a7920c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 -- 172.21.15.26:0/3302278712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a23a52e510 con 0x55a23a78a9c0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 0x55a23a7920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb854de640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbb70012700 tx=0x7fbb70012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37102c88b177b662 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb697fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70004030 con 0x55a23a78a9c0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fbb700041d0 con 0x55a23a78a9c0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb78036040 con 0x55a23a78a9c0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fbb700043d0 con 0x55a23a78a9c0 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb8ac56740 10 client.16269 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:10.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a23a3c1670 con 0x55a23a78a9c0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.413+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb78036040 con 0x55a23a78a9c0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fbb700039c0 con 0x55a23a78a9c0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 10 client.16269 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 10 client.16269 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 20 client.16269 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a7a52c0 0x55a23a7a7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 4 client.16269 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a23a6fcd90 con 0x55a23a7a52c0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a7a52c0 0x55a23a7a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a7a52c0 0x55a23a7a7730 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8a5285de3a307a server_cookie=401401d1c2b80362 in_seq=0 out_seq=0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.414+0000 7fbb697fa640 10 client.16269 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb697fa640 1 -- 172.21.15.26:0/3302278712 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a23a6fcd90 con 0x55a23a7a52c0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb697fa640 10 client.16269 handle_command_reply: tid=0 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 client.16269 shutdown 2024-04-23T18:22:10.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 msgr2=0x55a23a792600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbb70012700 tx=0x7fbb70012bc0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 shutdown_connections 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a7a52c0 0x55a23a7a7730 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a795090 0x55a23a792b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a792600 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb697fa640 0 client.16269 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3302278712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a78a0a0 0x55a23a7920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 >> 172.21.15.26:0/3302278712 conn(0x55a23a7863c0 msgr2=0x55a23a78c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 shutdown_connections 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3302278712 wait complete. 2024-04-23T18:22:10.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.415+0000 7fbb8ac56740 20 client.16269 trim_cache size 0 max 16384 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 Processor -- start 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 -- start start 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a79d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a792790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801b0430 con 0x55a23a78a0a0 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801adca0 con 0x55a23a795090 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801abaf0 con 0x55a23a78a9c0 2024-04-23T18:22:10.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a792790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb854de640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a79d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb85cdf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50758/0 (socket says 172.21.15.26:50758) 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb85cdf640 1 -- 172.21.15.26:0/3489505037 learned_addr learned my addr 172.21.15.26:0/3489505037 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 -- 172.21.15.26:0/3489505037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 msgr2=0x55a23a79d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a79d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 -- 172.21.15.26:0/3489505037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 msgr2=0x55a23a7a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb864e0640 1 -- 172.21.15.26:0/3489505037 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a23a7a3e90 con 0x55a23a795090 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.434+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb864e0640 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a792790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802ade0 tx=0x7fbb78032eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6ed3b0765cb4335 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb6b7fe640 1 -- 172.21.15.26:0/3489505037 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7804f040 con 0x55a23a795090 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb6b7fe640 1 -- 172.21.15.26:0/3489505037 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb78036040 con 0x55a23a795090 2024-04-23T18:22:10.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb6b7fe640 1 -- 172.21.15.26:0/3489505037 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7802fce0 con 0x55a23a795090 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3489505037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 msgr2=0x55a23a792790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a792790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802ade0 tx=0x7fbb78032eb0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3489505037 shutdown_connections 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a792790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a9c0 0x55a23a79d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3489505037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a0a0 0x55a23a7a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3489505037 >> 172.21.15.26:0/3489505037 conn(0x55a23a7863c0 msgr2=0x55a23a792080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.435+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3489505037 shutdown_connections 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3489505037 wait complete. 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 -- start start 2024-04-23T18:22:10.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 Processor -- start 2024-04-23T18:22:10.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 0x55a23a8092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a803870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801ac160 con 0x55a23a78a9c0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801ad5d0 con 0x55a23a795090 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801ac830 con 0x55a23a78a0a0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb8ac56740 10 client.0 _ll_register_callbacks cb 0x7fbb70009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb854de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb854de640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50770/0 (socket says 172.21.15.26:50770) 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.436+0000 7fbb854de640 1 -- 172.21.15.26:0/3209068759 learned_addr learned my addr 172.21.15.26:0/3209068759 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 0x55a23a8092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb864e0640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a803870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 -- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 msgr2=0x55a23a803870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a803870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 -- 172.21.15.26:0/3209068759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 msgr2=0x55a23a809810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 -- 172.21.15.26:0/3209068759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a23a803f60 con 0x55a23a78a0a0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb854de640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb85cdf640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 0x55a23a8092d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c09c8a0 tx=0x7fbb7c00a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1c6260528be3f93 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb697fa640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c00a250 con 0x55a23a78a0a0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fbb7c0bd0b0 con 0x55a23a78a0a0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb78036040 con 0x55a23a78a0a0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fbb7c09cba0 con 0x55a23a78a0a0 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb8ac56740 10 client.25693 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:10.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.437+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a23a798090 con 0x55a23a78a0a0 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb78036040 con 0x55a23a78a0a0 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fbb7c099210 con 0x55a23a78a0a0 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 10 client.25693 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 10 client.25693 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 20 client.25693 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a798800 0x55a23a79ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 4 client.25693 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await_for": 42.0}] 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await_for": 42.0}) -- 0x55a23a6f6c00 con 0x55a23a798800 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.438+0000 7fbb854de640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a798800 0x55a23a79ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.439+0000 7fbb697fa640 10 client.25693 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:10.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.439+0000 7fbb854de640 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a798800 0x55a23a79ac20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ce79133da466ac server_cookie=90cb1ac3c96a68b9 in_seq=0 out_seq=0 2024-04-23T18:22:10.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb697fa640 1 -- 172.21.15.26:0/3209068759 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a23a6f6c00 con 0x55a23a798800 2024-04-23T18:22:10.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb697fa640 10 client.25693 handle_command_reply: tid=0 2024-04-23T18:22:10.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 client.25693 shutdown 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 msgr2=0x55a23a8092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 0x55a23a8092d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c09c8a0 tx=0x7fbb7c00a440 comp rx=0 tx=0).stop 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 shutdown_connections 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a23a798800 0x55a23a79ac20 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a23a795090 0x55a23a803870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a23a78a9c0 0x55a23a809810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 --2- 172.21.15.26:0/3209068759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a23a78a0a0 0x55a23a8092d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb697fa640 0 client.25693 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 >> 172.21.15.26:0/3209068759 conn(0x55a23a7863c0 msgr2=0x55a23a78b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 shutdown_connections 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 1 -- 172.21.15.26:0/3209068759 wait complete. 2024-04-23T18:22:10.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.491+0000 7fbb8ac56740 20 client.25693 trim_cache size 0 max 16384 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 33, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 26, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "dd9551a1": { 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 26, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:22:10.487 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 81.1, 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:10.488 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 msgr2=0x7fbb6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 0x7fbb6c0a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802aeb0 tx=0x7fbb7803b000 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 msgr2=0x7fbb80198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00e9f0 tx=0x7fbb7c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 shutdown_connections 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbb6c0a3d20 0x7fbb6c0a61e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb8019d000 0x7fbb8019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb80107280 0x7fbb80198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 --2- 172.21.15.26:0/108939441 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbb80102240 0x7fbb80198790 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 >> 172.21.15.26:0/108939441 conn(0x7fbb800790e0 msgr2=0x7fbb80100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:10.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 shutdown_connections 2024-04-23T18:22:10.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:10.492+0000 7fbb87f43640 1 -- 172.21.15.26:0/108939441 wait complete. 2024-04-23T18:22:10.497 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dd9551a1', quiesce took 0.6 seconds. Will release after: 21.1 2024-04-23T18:22:10.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.75 MB/sec execute 125 sec latency 21667.259 ms 2024-04-23T18:22:11.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:10 smithi110 ceph-mon[32585]: pgmap v154: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:11.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:10 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/601019782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:11.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:10 smithi026 ceph-mon[29364]: pgmap v154: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:11.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:10 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/601019782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:11.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:10 smithi131 ceph-mon[33041]: pgmap v154: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-04-23T18:22:11.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:10 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/601019782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:11.671 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.69 MB/sec execute 126 sec latency 1215.117 ms 2024-04-23T18:22:12.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.64 MB/sec execute 127 sec latency 2215.257 ms 2024-04-23T18:22:13.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:12 smithi110 ceph-mon[32585]: pgmap v155: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.4 MiB/s wr, 117 op/s 2024-04-23T18:22:13.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:12 smithi026 ceph-mon[29364]: pgmap v155: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.4 MiB/s wr, 117 op/s 2024-04-23T18:22:13.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:12 smithi131 ceph-mon[33041]: pgmap v155: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.4 MiB/s wr, 117 op/s 2024-04-23T18:22:13.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.59 MB/sec execute 128 sec latency 3215.401 ms 2024-04-23T18:22:14.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.54 MB/sec execute 129 sec latency 4215.501 ms 2024-04-23T18:22:15.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:14 smithi110 ceph-mon[32585]: pgmap v156: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-04-23T18:22:15.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:14 smithi026 ceph-mon[29364]: pgmap v156: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-04-23T18:22:15.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:14 smithi131 ceph-mon[33041]: pgmap v156: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-04-23T18:22:15.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.49 MB/sec execute 130 sec latency 5215.538 ms 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 -- 172.21.15.26:0/3577602184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481061b0 msgr2=0x7f5d4810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/3577602184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481061b0 0x7f5d4810dd80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009ca0 tx=0x7f5d4002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 -- 172.21.15.26:0/3577602184 shutdown_connections 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/3577602184 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481061b0 0x7f5d4810dd80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/3577602184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d48101a90 0x7f5d48105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/3577602184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481010e0 0x7f5d481014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 -- 172.21.15.26:0/3577602184 >> 172.21.15.26:0/3577602184 conn(0x7f5d480771d0 msgr2=0x7f5d480775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 -- 172.21.15.26:0/3577602184 shutdown_connections 2024-04-23T18:22:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.715+0000 7f5d4dc23640 1 -- 172.21.15.26:0/3577602184 wait complete. 2024-04-23T18:22:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 Processor -- start 2024-04-23T18:22:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 -- start start 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 0x7f5d48102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481061b0 0x7f5d481030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48111100 con 0x7f5d481035f0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48110f80 con 0x7f5d481061b0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.716+0000 7f5d4dc23640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48111280 con 0x7f5d481010e0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50790/0 (socket says 172.21.15.26:50790) 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 -- 172.21.15.26:0/4041847630 learned_addr learned my addr 172.21.15.26:0/4041847630 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d477fe640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 0x7f5d48102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481061b0 0x7f5d481030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 -- 172.21.15.26:0/4041847630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 msgr2=0x7f5d48102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 0x7f5d48102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 -- 172.21.15.26:0/4041847630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481061b0 msgr2=0x7f5d481030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481061b0 0x7f5d481030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 -- 172.21.15.26:0/4041847630 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d481a6e40 con 0x7f5d481035f0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d477fe640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 0x7f5d48102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d47fff640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005cc0 tx=0x7f5d4002f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d44ff9640 1 -- 172.21.15.26:0/4041847630 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40003cf0 con 0x7f5d481035f0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d481a70d0 con 0x7f5d481035f0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d44ff9640 1 -- 172.21.15.26:0/4041847630 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d400044f0 con 0x7f5d481035f0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d44ff9640 1 -- 172.21.15.26:0/4041847630 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4002fd10 con 0x7f5d481035f0 2024-04-23T18:22:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.717+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d481a7620 con 0x7f5d481035f0 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.720+0000 7f5d44ff9640 1 -- 172.21.15.26:0/4041847630 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d40004040 con 0x7f5d481035f0 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.720+0000 7f5d50a12740 1 Processor -- start 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.720+0000 7f5d50a12740 1 -- start start 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985f40 0x560fbb9902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb992bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560fbb8ea970 con 0x560fbb985620 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fbb8874a0 con 0x560fbb990800 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d50a12740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560fbb8e6930 con 0x560fbb985f40 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50804/0 (socket says 172.21.15.26:50804) 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 -- 172.21.15.26:0/3977544638 learned_addr learned my addr 172.21.15.26:0/3977544638 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d44ff9640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 0x7f5d300a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d477fe640 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985f40 0x560fbb9902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d47fff640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 0x7f5d300a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 -- 172.21.15.26:0/3977544638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985f40 msgr2=0x560fbb9902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985f40 0x560fbb9902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 -- 172.21.15.26:0/3977544638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 msgr2=0x560fbb992bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb992bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 -- 172.21.15.26:0/3977544638 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fbb5fd210 con 0x560fbb985620 2024-04-23T18:22:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d44ff9640 1 -- 172.21.15.26:0/4041847630 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5d400f1ff0 con 0x7f5d481035f0 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.721+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5d38004650 tx=0x7f5d38010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6362a7d03665a050 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d47fff640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 0x7f5d300a6280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d400baba0 tx=0x7f5d400b7040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d2dffb640 1 -- 172.21.15.26:0/3977544638 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38016030 con 0x560fbb985620 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d2dffb640 1 -- 172.21.15.26:0/3977544638 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d38012930 con 0x560fbb985620 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d2dffb640 1 -- 172.21.15.26:0/3977544638 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38011750 con 0x560fbb985620 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 -- 172.21.15.26:0/3977544638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 msgr2=0x560fbb985a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5d38004650 tx=0x7f5d38010740 comp rx=0 tx=0).stop 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 -- 172.21.15.26:0/3977544638 shutdown_connections 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb992bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985f40 0x560fbb9902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 --2- 172.21.15.26:0/3977544638 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985620 0x560fbb985a00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 -- 172.21.15.26:0/3977544638 >> 172.21.15.26:0/3977544638 conn(0x560fbb981b30 msgr2=0x560fbb981f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 -- 172.21.15.26:0/3977544638 shutdown_connections 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.722+0000 7f5d50a12740 1 -- 172.21.15.26:0/3977544638 wait complete. 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 -- start start 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 Processor -- start 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb9901b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb98da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560fbb993430 con 0x560fbb985f40 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fbb8850b0 con 0x560fbb985620 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560fbb9935b0 con 0x560fbb990800 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d50a12740 10 client.0 _ll_register_callbacks cb 0x560fbb872d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50814/0 (socket says 172.21.15.26:50814) 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.723+0000 7f5d477fe640 1 -- 172.21.15.26:0/166730280 learned_addr learned my addr 172.21.15.26:0/166730280 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb9901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb98da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 -- 172.21.15.26:0/166730280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 msgr2=0x560fbb98da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb98da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 -- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 msgr2=0x560fbb9901b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb9901b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 -- 172.21.15.26:0/166730280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fbb6c9620 con 0x560fbb985f40 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb98da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d477fe640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c011a60 tx=0x7f5d3c011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44db6e3230d2bdae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c004460 con 0x560fbb985f40 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5d3c00f9a0 con 0x560fbb985f40 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d38012930 con 0x560fbb985f40 2024-04-23T18:22:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5d3c00fba0 con 0x560fbb985f40 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d50a12740 10 client.16287 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.724+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560fbb99f400 con 0x560fbb985f40 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d38012930 con 0x560fbb985f40 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f5d3c018070 con 0x560fbb985f40 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 10 client.16287 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 10 client.16287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 20 client.16287 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbb9a0980 0x560fbb9a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 4 client.16287 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fbb8f8390 con 0x560fbb9a0980 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.725+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbb9a0980 0x560fbb9a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d26ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbb9a0980 0x560fbb9a2d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8d885c17b23736 server_cookie=142c57180562cb48 in_seq=0 out_seq=0 2024-04-23T18:22:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d26ffd640 1 -- 172.21.15.26:0/166730280 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fbb8f8390 con 0x560fbb9a0980 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d26ffd640 10 client.16287 handle_command_reply: tid=0 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d50a12740 1 client.16287 shutdown 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 msgr2=0x560fbb98d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.726+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c011a60 tx=0x7f5d3c011f20 comp rx=0 tx=0).stop 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 shutdown_connections 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbb9a0980 0x560fbb9a2d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb98da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb98d4a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d26ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 --2- 172.21.15.26:0/166730280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb9901b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 >> 172.21.15.26:0/166730280 conn(0x560fbb981b30 msgr2=0x560fbb9878b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 shutdown_connections 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 1 -- 172.21.15.26:0/166730280 wait complete. 2024-04-23T18:22:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.727+0000 7f5d50a12740 20 client.16287 trim_cache size 0 max 16384 2024-04-23T18:22:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 Processor -- start 2024-04-23T18:22:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 -- start start 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 0x560fbb98fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb998d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481b24e0 con 0x560fbb985f40 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481afd50 con 0x560fbb990800 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d50a12740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481adbc0 con 0x560fbb985620 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50826/0 (socket says 172.21.15.26:50826) 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 -- 172.21.15.26:0/450576701 learned_addr learned my addr 172.21.15.26:0/450576701 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 0x560fbb98fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 -- 172.21.15.26:0/450576701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 msgr2=0x560fbb98fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 0x560fbb98fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 -- 172.21.15.26:0/450576701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 msgr2=0x560fbb998d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb998d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d477fe640 1 -- 172.21.15.26:0/450576701 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fbb985d70 con 0x560fbb985f40 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.742+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 0x560fbb98fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d477fe640 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c004e10 tx=0x7f5d3c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9210b076c5f21c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d2d7fa640 1 -- 172.21.15.26:0/450576701 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c026040 con 0x560fbb985f40 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d2d7fa640 1 -- 172.21.15.26:0/450576701 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d3c005340 con 0x560fbb985f40 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d2d7fa640 1 -- 172.21.15.26:0/450576701 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c004800 con 0x560fbb985f40 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 -- 172.21.15.26:0/450576701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 msgr2=0x560fbb998800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c004e10 tx=0x7f5d3c005fd0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 -- 172.21.15.26:0/450576701 shutdown_connections 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb990800 0x560fbb998d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb998800 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 --2- 172.21.15.26:0/450576701 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb985620 0x560fbb98fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 -- 172.21.15.26:0/450576701 >> 172.21.15.26:0/450576701 conn(0x560fbb981b30 msgr2=0x560fbb99b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 -- 172.21.15.26:0/450576701 shutdown_connections 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.743+0000 7f5d50a12740 1 -- 172.21.15.26:0/450576701 wait complete. 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 -- start start 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 Processor -- start 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb996490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb996f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ae210 con 0x560fbb985f40 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481af680 con 0x560fbb985620 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5d481ae8e0 con 0x560fbb990800 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d50a12740 10 client.0 _ll_register_callbacks cb 0x7f5d3c008d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d477fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:50834/0 (socket says 172.21.15.26:50834) 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d477fe640 1 -- 172.21.15.26:0/1764092333 learned_addr learned my addr 172.21.15.26:0/1764092333 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb996490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.744+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb996f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 -- 172.21.15.26:0/1764092333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 msgr2=0x560fbb996f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb996f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 -- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 msgr2=0x560fbb996490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb996490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 -- 172.21.15.26:0/1764092333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fbba06150 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb996f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d477fe640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00f880 tx=0x7f5d3c004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a3c116e12865cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c039060 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5d3c004e40 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d3c005340 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5d3c030040 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d50a12740 10 client.16293 fetch_fsmap learned FSMap version 33 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560fbba073d0 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.745+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d3c005340 con 0x560fbb985f40 2024-04-23T18:22:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f5d3c033070 con 0x560fbb985f40 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 10 client.16293 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 10 client.16293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 20 client.16293 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbba07a90 0x560fbba09eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 4 client.16293 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560fbb8f2200 con 0x560fbba07a90 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbba07a90 0x560fbba09eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d46ffd640 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbba07a90 0x560fbba09eb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72203fa2fde48b6d server_cookie=a5dbfdb4046d295a in_seq=0 out_seq=0 2024-04-23T18:22:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.746+0000 7f5d26ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.762+0000 7f5d26ffd640 1 -- 172.21.15.26:0/1764092333 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560fbb8f2200 con 0x560fbba07a90 2024-04-23T18:22:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.762+0000 7f5d26ffd640 10 client.16293 handle_command_reply: tid=0 2024-04-23T18:22:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.762+0000 7f5d50a12740 1 client.16293 shutdown 2024-04-23T18:22:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.762+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 msgr2=0x560fbb9969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.762+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00f880 tx=0x7f5d3c004330 comp rx=0 tx=0).stop 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 shutdown_connections 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560fbba07a90 0x560fbba09eb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560fbb990800 0x560fbb996f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560fbb985f40 0x560fbb9969d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d26ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 --2- 172.21.15.26:0/1764092333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fbb985620 0x560fbb996490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 >> 172.21.15.26:0/1764092333 conn(0x560fbb981b30 msgr2=0x560fbb99bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 shutdown_connections 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 1 -- 172.21.15.26:0/1764092333 wait complete. 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d50a12740 20 client.16293 trim_cache size 0 max 16384 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:22:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:22:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:22:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:22:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 msgr2=0x7f5d300a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 0x7f5d300a6280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d400baba0 tx=0x7f5d400b7040 comp rx=0 tx=0).stop 2024-04-23T18:22:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 msgr2=0x7f5d481a67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005cc0 tx=0x7f5d4002f470 comp rx=0 tx=0).stop 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 shutdown_connections 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5d300a3dc0 0x7f5d300a6280 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d481061b0 0x7f5d481030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5d481035f0 0x7f5d481a67a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 --2- 172.21.15.26:0/4041847630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5d481010e0 0x7f5d48102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.763+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 >> 172.21.15.26:0/4041847630 conn(0x7f5d480771d0 msgr2=0x7f5d4806a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.764+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 shutdown_connections 2024-04-23T18:22:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:15.764+0000 7f5d4dc23640 1 -- 172.21.15.26:0/4041847630 wait complete. 2024-04-23T18:22:16.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.44 MB/sec execute 131 sec latency 6215.678 ms 2024-04-23T18:22:17.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:16 smithi110 ceph-mon[32585]: pgmap v157: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-04-23T18:22:17.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:16 smithi026 ceph-mon[29364]: pgmap v157: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-04-23T18:22:17.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:16 smithi131 ceph-mon[33041]: pgmap v157: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-04-23T18:22:17.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.39 MB/sec execute 132 sec latency 7215.797 ms 2024-04-23T18:22:18.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.34 MB/sec execute 133 sec latency 8215.919 ms 2024-04-23T18:22:19.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:18 smithi110 ceph-mon[32585]: pgmap v158: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:19.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:18 smithi026 ceph-mon[29364]: pgmap v158: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:19.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:18 smithi131 ceph-mon[33041]: pgmap v158: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:19.672 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.29 MB/sec execute 134 sec latency 9215.983 ms 2024-04-23T18:22:20.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:20.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:19 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:22:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:19 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:22:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:19 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:22:20.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.25 MB/sec execute 135 sec latency 10216.087 ms 2024-04-23T18:22:21.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:20 smithi110 ceph-mon[32585]: pgmap v159: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:20 smithi026 ceph-mon[29364]: pgmap v159: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:20 smithi131 ceph-mon[33041]: pgmap v159: 97 pgs: 97 active+clean; 162 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-23T18:22:21.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.20 MB/sec execute 136 sec latency 11216.254 ms 2024-04-23T18:22:22.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.16 MB/sec execute 137 sec latency 12216.385 ms 2024-04-23T18:22:23.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:22 smithi110 ceph-mon[32585]: pgmap v160: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 2.2 MiB/s wr, 99 op/s 2024-04-23T18:22:23.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:22 smithi026 ceph-mon[29364]: pgmap v160: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 2.2 MiB/s wr, 99 op/s 2024-04-23T18:22:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:22 smithi131 ceph-mon[33041]: pgmap v160: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 2.2 MiB/s wr, 99 op/s 2024-04-23T18:22:23.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.11 MB/sec execute 138 sec latency 13216.473 ms 2024-04-23T18:22:24.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.07 MB/sec execute 139 sec latency 14216.547 ms 2024-04-23T18:22:25.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:24 smithi110 ceph-mon[32585]: pgmap v161: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-23T18:22:25.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:24 smithi026 ceph-mon[29364]: pgmap v161: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-23T18:22:25.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:24 smithi131 ceph-mon[33041]: pgmap v161: 97 pgs: 97 active+clean; 163 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-04-23T18:22:25.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 6.02 MB/sec execute 140 sec latency 15216.651 ms 2024-04-23T18:22:26.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.98 MB/sec execute 141 sec latency 16216.762 ms 2024-04-23T18:22:27.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:26 smithi110 ceph-mon[32585]: pgmap v162: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-04-23T18:22:27.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:26 smithi026 ceph-mon[29364]: pgmap v162: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-04-23T18:22:27.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:26 smithi131 ceph-mon[33041]: pgmap v162: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-04-23T18:22:27.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.94 MB/sec execute 142 sec latency 17216.897 ms 2024-04-23T18:22:28.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.90 MB/sec execute 143 sec latency 18217.005 ms 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 -- 172.21.15.26:0/437264197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0074c10 msgr2=0x7fb0c0112a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 --2- 172.21.15.26:0/437264197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0074c10 0x7fb0c0112a20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009900 tx=0x7fb0b002a480 comp rx=0 tx=0).stop 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 -- 172.21.15.26:0/437264197 shutdown_connections 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 --2- 172.21.15.26:0/437264197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c0112f60 0x7fb0c0115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 --2- 172.21.15.26:0/437264197 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0074c10 0x7fb0c0112a20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 --2- 172.21.15.26:0/437264197 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c00742f0 0x7fb0c00746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.863+0000 7fb0c7279640 1 -- 172.21.15.26:0/437264197 >> 172.21.15.26:0/437264197 conn(0x7fb0c00fda20 msgr2=0x7fb0c00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 -- 172.21.15.26:0/437264197 shutdown_connections 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 -- 172.21.15.26:0/437264197 wait complete. 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 Processor -- start 2024-04-23T18:22:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 -- start start 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c00742f0 0x7fb0c01a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c0074c10 0x7fb0c01a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.864+0000 7fb0c7279640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01187d0 con 0x7fb0c0112f60 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c7279640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c0118650 con 0x7fb0c00742f0 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c7279640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c0118950 con 0x7fb0c0074c10 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40912/0 (socket says 172.21.15.26:40912) 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 -- 172.21.15.26:0/1365087270 learned_addr learned my addr 172.21.15.26:0/1365087270 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c4814640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c0074c10 0x7fb0c01a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5015640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c00742f0 0x7fb0c01a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 -- 172.21.15.26:0/1365087270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c0074c10 msgr2=0x7fb0c01a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c0074c10 0x7fb0c01a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 -- 172.21.15.26:0/1365087270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c00742f0 msgr2=0x7fb0c01a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c00742f0 0x7fb0c01a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 -- 172.21.15.26:0/1365087270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c019f430 con 0x7fb0c0112f60 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0c5816640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800e9d0 tx=0x7fb0b800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.865+0000 7fb0b67fc640 1 -- 172.21.15.26:0/1365087270 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b800cd80 con 0x7fb0c0112f60 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.866+0000 7fb0b67fc640 1 -- 172.21.15.26:0/1365087270 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b8005020 con 0x7fb0c0112f60 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.866+0000 7fb0b67fc640 1 -- 172.21.15.26:0/1365087270 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80116d0 con 0x7fb0c0112f60 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.866+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c019f720 con 0x7fb0c0112f60 2024-04-23T18:22:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.866+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c01ab9e0 con 0x7fb0c0112f60 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.868+0000 7fb0b67fc640 1 -- 172.21.15.26:0/1365087270 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb0b80051d0 con 0x7fb0c0112f60 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.868+0000 7fb0c9f8d740 1 Processor -- start 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.868+0000 7fb0c9f8d740 1 -- start start 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57db4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dbf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x563a57d19970 con 0x563a57dbf800 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563a57cb64a0 con 0x563a57db4f40 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x563a57d15930 con 0x563a57db4620 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40918/0 (socket says 172.21.15.26:40918) 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 -- 172.21.15.26:0/2165887890 learned_addr learned my addr 172.21.15.26:0/2165887890 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57db4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c5015640 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dbf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0b67fc640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 0x7fb0ac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 -- 172.21.15.26:0/2165887890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 msgr2=0x563a57db4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57db4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 -- 172.21.15.26:0/2165887890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 msgr2=0x563a57dbf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dbf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 -- 172.21.15.26:0/2165887890 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a57a2c210 con 0x563a57dbf800 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c5816640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 0x7fb0ac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0c4814640 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009900 tx=0x7fb0b0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cb87f6916a34e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.869+0000 7fb0b67fc640 1 -- 172.21.15.26:0/1365087270 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0b80d0840 con 0x7fb0c0112f60 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0ab7fe640 1 -- 172.21.15.26:0/2165887890 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0044020 con 0x563a57dbf800 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0ab7fe640 1 -- 172.21.15.26:0/2165887890 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b0004030 con 0x563a57dbf800 2024-04-23T18:22:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0ab7fe640 1 -- 172.21.15.26:0/2165887890 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035070 con 0x563a57dbf800 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/2165887890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 msgr2=0x563a57dc1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c5816640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 0x7fb0ac0a6300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ce5c0 tx=0x7fb0b801d000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009900 tx=0x7fb0b0005c00 comp rx=0 tx=0).stop 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/2165887890 shutdown_connections 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dbf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc1bf0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/2165887890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57db4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/2165887890 >> 172.21.15.26:0/2165887890 conn(0x563a57db0b30 msgr2=0x563a57db0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.870+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/2165887890 shutdown_connections 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/2165887890 wait complete. 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 -- start start 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 Processor -- start 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dbc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57dbf800 0x563a57dbc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x563a57dc23d0 con 0x563a57db4620 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563a57cb40b0 con 0x563a57dbf800 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x563a57dc2550 con 0x563a57db4f40 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c9f8d740 10 client.0 _ll_register_callbacks cb 0x563a57ca1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.871+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40924/0 (socket says 172.21.15.26:40924) 2024-04-23T18:22:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 -- 172.21.15.26:0/871271631 learned_addr learned my addr 172.21.15.26:0/871271631 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c5015640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dbc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57dbf800 0x563a57dbc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 -- 172.21.15.26:0/871271631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 msgr2=0x563a57dbc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dbc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 -- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57dbf800 msgr2=0x563a57dbc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57dbf800 0x563a57dbc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 -- 172.21.15.26:0/871271631 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a57af8620 con 0x563a57db4620 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c4814640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0006010 tx=0x7fb0b0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37154246cd103953 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035070 con 0x563a57db4620 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0b0005050 con 0x563a57db4620 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.872+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b0004030 con 0x563a57db4620 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0b0044070 con 0x563a57db4620 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 10 client.16305 fetch_fsmap learned FSMap version 34 2024-04-23T18:22:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a57dce320 con 0x563a57db4620 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b0004030 con 0x563a57db4620 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7fb0b00046c0 con 0x563a57db4620 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 10 client.16305 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 20 client.16305 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57dcf670 0x563a57dd1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 4 client.16305 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a57d27390 con 0x563a57dcf670 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.873+0000 7fb0c5015640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57dcf670 0x563a57dd1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.874+0000 7fb0a8ff9640 10 client.16305 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.874+0000 7fb0c5015640 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57dcf670 0x563a57dd1a60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cd7ac6353907727 server_cookie=c1ccafba431a510b in_seq=0 out_seq=0 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.874+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/871271631 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a57d27390 con 0x563a57dcf670 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.874+0000 7fb0a8ff9640 10 client.16305 handle_command_reply: tid=0 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.874+0000 7fb0c9f8d740 1 client.16305 shutdown 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 msgr2=0x563a57dbf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0006010 tx=0x7fb0b0031400 comp rx=0 tx=0).stop 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 shutdown_connections 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57dcf670 0x563a57dd1a60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57dbf800 0x563a57dbc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dbc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/871271631 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57db4620 0x563a57dbf130 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0a8ff9640 0 client.16305 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 >> 172.21.15.26:0/871271631 conn(0x563a57db0b30 msgr2=0x563a57db6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 shutdown_connections 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/871271631 wait complete. 2024-04-23T18:22:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.875+0000 7fb0c9f8d740 20 client.16305 trim_cache size 0 max 16384 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 Processor -- start 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 -- start start 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57dcb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dafcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b6970 con 0x563a57dbf800 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b41e0 con 0x563a57db4f40 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b2030 con 0x563a57db4620 2024-04-23T18:22:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40928/0 (socket says 172.21.15.26:40928) 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c4814640 1 -- 172.21.15.26:0/912086935 learned_addr learned my addr 172.21.15.26:0/912086935 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.890+0000 7fb0c5015640 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dafcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57dcb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 -- 172.21.15.26:0/912086935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 msgr2=0x563a57dcb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57dcb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 -- 172.21.15.26:0/912086935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 msgr2=0x563a57dafcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dafcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 -- 172.21.15.26:0/912086935 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a57db4d70 con 0x563a57dbf800 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0c4814640 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb0b002a9a0 tx=0x7fb0b002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cccb1e21233b8c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0aaffd640 1 -- 172.21.15.26:0/912086935 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035070 con 0x563a57dbf800 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0aaffd640 1 -- 172.21.15.26:0/912086935 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b005d040 con 0x563a57dbf800 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.891+0000 7fb0aaffd640 1 -- 172.21.15.26:0/912086935 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b005c040 con 0x563a57dbf800 2024-04-23T18:22:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/912086935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 msgr2=0x563a57dd7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb0b002a9a0 tx=0x7fb0b002a480 comp rx=0 tx=0).stop 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/912086935 shutdown_connections 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4f40 0x563a57dafcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dd7060 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/912086935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4620 0x563a57dcb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/912086935 >> 172.21.15.26:0/912086935 conn(0x563a57db0b30 msgr2=0x563a57dbbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/912086935 shutdown_connections 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/912086935 wait complete. 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 -- start start 2024-04-23T18:22:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.892+0000 7fb0c9f8d740 1 Processor -- start 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4620 0x563a57dc7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dc5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b26a0 con 0x563a57dbf800 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b3b10 con 0x563a57db4620 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01b2d70 con 0x563a57db4f40 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c9f8d740 10 client.0 _ll_register_callbacks cb 0x7fb0b00041e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40942/0 (socket says 172.21.15.26:40942) 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 -- 172.21.15.26:0/3414729707 learned_addr learned my addr 172.21.15.26:0/3414729707 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c5015640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dc5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4620 0x563a57dc7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 -- 172.21.15.26:0/3414729707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 msgr2=0x563a57dc5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dc5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 -- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4620 msgr2=0x563a57dc7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4620 0x563a57dc7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 -- 172.21.15.26:0/3414729707 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a57e348e0 con 0x563a57dbf800 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0c4814640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0004b10 tx=0x7fb0b0040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2ad8a95145324b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.893+0000 7fb0a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035070 con 0x563a57dbf800 2024-04-23T18:22:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b005d040 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0b0003700 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb0b005c070 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 10 client.16311 fetch_fsmap learned FSMap version 34 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a57e35a20 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b005d040 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7fb0b006d080 con 0x563a57dbf800 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 10 client.16311 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 10 client.16311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 20 client.16311 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57e35fb0 0x563a57e383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 4 client.16311 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a57d21200 con 0x563a57e35fb0 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.894+0000 7fb0c5015640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57e35fb0 0x563a57e383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.895+0000 7fb0a8ff9640 10 client.16311 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.895+0000 7fb0c5015640 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57e35fb0 0x563a57e383d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47da8a2619deb149 server_cookie=208de90d0747a973 in_seq=0 out_seq=0 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.906+0000 7fb0a8ff9640 1 -- 172.21.15.26:0/3414729707 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a57d21200 con 0x563a57e35fb0 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.906+0000 7fb0a8ff9640 10 client.16311 handle_command_reply: tid=0 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.906+0000 7fb0c9f8d740 1 client.16311 shutdown 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 msgr2=0x563a57dc5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0004b10 tx=0x7fb0b0040040 comp rx=0 tx=0).stop 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 shutdown_connections 2024-04-23T18:22:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x563a57e35fb0 0x563a57e383d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x563a57dbf800 0x563a57dc5770 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0a8ff9640 0 client.16311 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563a57db4f40 0x563a57dc5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 --2- 172.21.15.26:0/3414729707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a57db4620 0x563a57dc7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 >> 172.21.15.26:0/3414729707 conn(0x563a57db0b30 msgr2=0x563a57dbc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 shutdown_connections 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 1 -- 172.21.15.26:0/3414729707 wait complete. 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c9f8d740 20 client.16311 trim_cache size 0 max 16384 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:22:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:22:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:22:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.907+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 msgr2=0x7fb0ac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 0x7fb0ac0a6300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ce5c0 tx=0x7fb0b801d000 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 msgr2=0x7fb0c019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800e9d0 tx=0x7fb0b800ee90 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 shutdown_connections 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb0ac0a3e40 0x7fb0ac0a6300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb0c0112f60 0x7fb0c019eb70 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0c0074c10 0x7fb0c01a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 --2- 172.21.15.26:0/1365087270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c00742f0 0x7fb0c01a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 >> 172.21.15.26:0/1365087270 conn(0x7fb0c00fda20 msgr2=0x7fb0c00ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 shutdown_connections 2024-04-23T18:22:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:28.908+0000 7fb0c7279640 1 -- 172.21.15.26:0/1365087270 wait complete. 2024-04-23T18:22:29.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:28 smithi110 ceph-mon[32585]: pgmap v163: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:29.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:28 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:28 smithi026 ceph-mon[29364]: pgmap v163: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:28 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:28 smithi131 ceph-mon[33041]: pgmap v163: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:28 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:29.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.86 MB/sec execute 144 sec latency 19217.106 ms 2024-04-23T18:22:30.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.82 MB/sec execute 145 sec latency 20217.216 ms 2024-04-23T18:22:31.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:30 smithi110 ceph-mon[32585]: pgmap v164: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:30 smithi026 ceph-mon[29364]: pgmap v164: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:31.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:30 smithi131 ceph-mon[33041]: pgmap v164: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 918 KiB/s wr, 45 op/s 2024-04-23T18:22:31.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:31.673 INFO:tasks.workunit.client.0.smithi026.stdout: 1 313393 5.78 MB/sec execute 146 sec latency 21217.295 ms 2024-04-23T18:22:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 -- 172.21.15.26:0/276419507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102bf0 msgr2=0x7f28bc106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 --2- 172.21.15.26:0/276419507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102bf0 0x7f28bc106ba0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009900 tx=0x7f28b002a410 comp rx=0 tx=0).stop 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 -- 172.21.15.26:0/276419507 shutdown_connections 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 --2- 172.21.15.26:0/276419507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 0x7f28bc10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 --2- 172.21.15.26:0/276419507 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102bf0 0x7f28bc106ba0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 --2- 172.21.15.26:0/276419507 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc102240 0x7f28bc102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 -- 172.21.15.26:0/276419507 >> 172.21.15.26:0/276419507 conn(0x7f28bc0790e0 msgr2=0x7f28bc0794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 -- 172.21.15.26:0/276419507 shutdown_connections 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 -- 172.21.15.26:0/276419507 wait complete. 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.719+0000 7f28c20e8640 1 Processor -- start 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 -- start start 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 0x7f28bc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 0x7f28bc1a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:31.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc1121f0 con 0x7f28bc102240 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc112070 con 0x7f28bc107280 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28c20e8640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc112370 con 0x7f28bc074cd0 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40956/0 (socket says 172.21.15.26:40956) 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 -- 172.21.15.26:0/4063300889 learned_addr learned my addr 172.21.15.26:0/4063300889 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28baffd640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 0x7f28bc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bbfff640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 0x7f28bc1a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 -- 172.21.15.26:0/4063300889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 msgr2=0x7f28bc1a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 0x7f28bc1a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 -- 172.21.15.26:0/4063300889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 msgr2=0x7f28bc074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 0x7f28bc074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bb7fe640 1 -- 172.21.15.26:0/4063300889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc1a7060 con 0x7f28bc102240 2024-04-23T18:22:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.720+0000 7f28bbfff640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 0x7f28bc1a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28bb7fe640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f28ac00d950 tx=0x7f28ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac017030 con 0x7f28bc102240 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28bc1a7350 con 0x7f28bc102240 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28ac004020 con 0x7f28bc102240 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0047a0 con 0x7f28bc102240 2024-04-23T18:22:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.721+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28bc1a78c0 con 0x7f28bc102240 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.722+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28bc0691c0 con 0x7f28bc102240 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.723+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28ac009a30 con 0x7f28bc102240 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.724+0000 7f28b8ff9640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 0x7f28a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.724+0000 7f28baffd640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 0x7f28a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.724+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f28ac0d02e0 con 0x7f28bc102240 2024-04-23T18:22:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.724+0000 7f28baffd640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 0x7f28a40a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f28b002aa00 tx=0x7f28b0031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:31.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.728+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28ac09c9b0 con 0x7f28bc102240 2024-04-23T18:22:31.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.954+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28bc10a4a0 con 0x7f28bc102240 2024-04-23T18:22:31.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.955+0000 7f28b8ff9640 1 -- 172.21.15.26:0/4063300889 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11888 (secure 0 0 0) 0x7f28ac096e20 con 0x7f28bc102240 2024-04-23T18:22:31.948 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:22:31.948 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:22:27.115453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:22:31.949 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 34 2024-04-23T18:22:31.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 msgr2=0x7f28a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:31.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 0x7f28a40a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f28b002aa00 tx=0x7f28b0031400 comp rx=0 tx=0).stop 2024-04-23T18:22:31.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 msgr2=0x7f28bc074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f28ac00d950 tx=0x7f28ac00de10 comp rx=0 tx=0).stop 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 shutdown_connections 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f28a40a3d20 0x7f28a40a61e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28bc107280 0x7f28bc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28bc074cd0 0x7f28bc1a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 --2- 172.21.15.26:0/4063300889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f28bc102240 0x7f28bc074250 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 >> 172.21.15.26:0/4063300889 conn(0x7f28bc0790e0 msgr2=0x7f28bc100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 shutdown_connections 2024-04-23T18:22:31.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:31.957+0000 7f28c20e8640 1 -- 172.21.15.26:0/4063300889 wait complete. 2024-04-23T18:22:31.961 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id dd9551a1 --release --await' 2024-04-23T18:22:31.962 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id dd9551a1 --release --await 2024-04-23T18:22:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 -- 172.21.15.26:0/1521478625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde8107270 msgr2=0x7fdde810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 --2- 172.21.15.26:0/1521478625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde8107270 0x7fdde810ee40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009ca0 tx=0x7fdde002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 -- 172.21.15.26:0/1521478625 shutdown_connections 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 --2- 172.21.15.26:0/1521478625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde8107270 0x7fdde810ee40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 --2- 172.21.15.26:0/1521478625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde806b1f0 0x7fdde8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 --2- 172.21.15.26:0/1521478625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde806a840 0x7fdde806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 -- 172.21.15.26:0/1521478625 >> 172.21.15.26:0/1521478625 conn(0x7fdde8077eb0 msgr2=0x7fdde80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 -- 172.21.15.26:0/1521478625 shutdown_connections 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.078+0000 7fddee449640 1 -- 172.21.15.26:0/1521478625 wait complete. 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 Processor -- start 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 -- start start 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 0x7fdde8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde819cf30 0x7fdde819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdde8112230 con 0x7fdde806a840 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81120b0 con 0x7fdde8107270 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddee449640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81123b0 con 0x7fdde819cf30 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40972/0 (socket says 172.21.15.26:40972) 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 -- 172.21.15.26:0/3492479203 learned_addr learned my addr 172.21.15.26:0/3492479203 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 0x7fdde8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fddec9e6640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde819cf30 0x7fdde819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 -- 172.21.15.26:0/3492479203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde819cf30 msgr2=0x7fdde819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde819cf30 0x7fdde819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 -- 172.21.15.26:0/3492479203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 msgr2=0x7fdde8198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 0x7fdde8198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde7fff640 1 -- 172.21.15.26:0/3492479203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde819da20 con 0x7fdde806a840 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.079+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 0x7fdde8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c920 tx=0x7fddd800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fdde57fa640 1 -- 172.21.15.26:0/3492479203 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8007c70 con 0x7fdde806a840 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fdde57fa640 1 -- 172.21.15.26:0/3492479203 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd8007e10 con 0x7fdde806a840 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fdde57fa640 1 -- 172.21.15.26:0/3492479203 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8018820 con 0x7fdde806a840 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde819dd10 con 0x7fdde806a840 2024-04-23T18:22:32.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.080+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde81a5500 con 0x7fdde806a840 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fdde57fa640 1 -- 172.21.15.26:0/3492479203 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fddd800f450 con 0x7fdde806a840 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 Processor -- start 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 -- start start 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c92870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560807bea6d0 con 0x560807c85490 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560807b871a0 con 0x560807c90480 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddf1234740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560807be6690 con 0x560807c85db0 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40986/0 (socket says 172.21.15.26:40986) 2024-04-23T18:22:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 -- 172.21.15.26:0/1876517006 learned_addr learned my addr 172.21.15.26:0/1876517006 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fdde57fa640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 0x7fddd00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fdde77fe640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c92870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fdde7fff640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 -- 172.21.15.26:0/1876517006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 msgr2=0x560807c8ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 -- 172.21.15.26:0/1876517006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 msgr2=0x560807c92870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c92870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fddec9e6640 1 -- 172.21.15.26:0/1876517006 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560807b915b0 con 0x560807c85490 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.083+0000 7fdde7fff640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fdde57fa640 1 -- 172.21.15.26:0/3492479203 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fddd80d09a0 con 0x7fdde806a840 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fddec9e6640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 0x7fddd00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fddec9e6640 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009a00 tx=0x7fdde0031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d84b3d070748fa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fddce7fc640 1 -- 172.21.15.26:0/1876517006 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0004040 con 0x560807c85490 2024-04-23T18:22:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fddce7fc640 1 -- 172.21.15.26:0/1876517006 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdde0036070 con 0x560807c85490 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.084+0000 7fddce7fc640 1 -- 172.21.15.26:0/1876517006 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde002faa0 con 0x560807c85490 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- 172.21.15.26:0/1876517006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 msgr2=0x560807c85870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddec9e6640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 0x7fddd00a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009dc0 tx=0x7fdde0048000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009a00 tx=0x7fdde0031af0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- 172.21.15.26:0/1876517006 shutdown_connections 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c92870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 --2- 172.21.15.26:0/1876517006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c85870 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- 172.21.15.26:0/1876517006 >> 172.21.15.26:0/1876517006 conn(0x560807c817b0 msgr2=0x560807c81b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- 172.21.15.26:0/1876517006 shutdown_connections 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- 172.21.15.26:0/1876517006 wait complete. 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 -- start start 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.085+0000 7fddf1234740 1 Processor -- start 2024-04-23T18:22:32.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c8d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x560807c93190 con 0x560807c85490 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560807b84b20 con 0x560807c90480 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x560807c93310 con 0x560807c85db0 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 10 client.0 _ll_register_callbacks cb 0x560807b84530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40990/0 (socket says 172.21.15.26:40990) 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 -- 172.21.15.26:0/3508670128 learned_addr learned my addr 172.21.15.26:0/3508670128 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:32.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddec9e6640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c8d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 -- 172.21.15.26:0/3508670128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 msgr2=0x560807c8d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 -- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 msgr2=0x560807c8d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c8d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 -- 172.21.15.26:0/3508670128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608078fc920 con 0x560807c85490 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddec9e6640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c8d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fdde7fff640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fddd809c670 tx=0x7fddd806c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab54e9bbfa298147 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd80a0d20 con 0x560807c85490 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fddd800fa90 con 0x560807c85490 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.086+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdde0036070 con 0x560807c85490 2024-04-23T18:22:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fddd80a55e0 con 0x560807c85490 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 10 client.16329 fetch_fsmap learned FSMap version 34 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5608079c8d30 con 0x560807c85490 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdde0036070 con 0x560807c85490 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7fddd80a5150 con 0x560807c85490 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 10 client.16329 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 10 client.16329 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 20 client.16329 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807ca0310 0x560807ca27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 4 client.16329 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560807ba8500 con 0x560807ca0310 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.087+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807ca0310 0x560807ca27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.088+0000 7fdde77fe640 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807ca0310 0x560807ca27b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6ebd0a08bf03f4f server_cookie=227b1a8076261150 in_seq=0 out_seq=0 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.088+0000 7fddc77fe640 10 client.16329 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.088+0000 7fddc77fe640 1 -- 172.21.15.26:0/3508670128 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560807ba8500 con 0x560807ca0310 2024-04-23T18:22:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.088+0000 7fddc77fe640 10 client.16329 handle_command_reply: tid=0 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.088+0000 7fddf1234740 1 client.16329 shutdown 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 msgr2=0x560807c8ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fddd809c670 tx=0x7fddd806c250 comp rx=0 tx=0).stop 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 shutdown_connections 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807ca0310 0x560807ca27b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c8d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85db0 0x560807c8d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddc77fe640 0 client.16329 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 --2- 172.21.15.26:0/3508670128 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85490 0x560807c8ff00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 >> 172.21.15.26:0/3508670128 conn(0x560807c817b0 msgr2=0x560807c875b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 shutdown_connections 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 1 -- 172.21.15.26:0/3508670128 wait complete. 2024-04-23T18:22:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.089+0000 7fddf1234740 20 client.16329 trim_cache size 0 max 16384 2024-04-23T18:22:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 Processor -- start 2024-04-23T18:22:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 -- start start 2024-04-23T18:22:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c85490 0x560807c9c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c90480 0x560807c8fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81b0320 con 0x560807c85db0 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81adb90 con 0x560807c85490 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fddf1234740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81aba00 con 0x560807c90480 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40992/0 (socket says 172.21.15.26:40992) 2024-04-23T18:22:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 -- 172.21.15.26:0/497936896 learned_addr learned my addr 172.21.15.26:0/497936896 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 -- 172.21.15.26:0/497936896 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c90480 msgr2=0x560807c8fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c90480 0x560807c8fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde77fe640 1 -- 172.21.15.26:0/497936896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c85490 msgr2=0x560807c9c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.106+0000 7fdde7fff640 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c85490 0x560807c9c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fdde77fe640 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c85490 0x560807c9c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fdde77fe640 1 -- 172.21.15.26:0/497936896 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560807c84aa0 con 0x560807c85db0 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fdde77fe640 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdddc000c00 tx=0x7fdddc00cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dd2c6c67ab1b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddcdffb640 1 -- 172.21.15.26:0/497936896 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc015ba0 con 0x560807c85db0 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddcdffb640 1 -- 172.21.15.26:0/497936896 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdddc015d40 con 0x560807c85db0 2024-04-23T18:22:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddcdffb640 1 -- 172.21.15.26:0/497936896 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0225f0 con 0x560807c85db0 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 -- 172.21.15.26:0/497936896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 msgr2=0x560807c87b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdddc000c00 tx=0x7fdddc00cfb0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 -- 172.21.15.26:0/497936896 shutdown_connections 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c90480 0x560807c8fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c87b80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 --2- 172.21.15.26:0/497936896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c85490 0x560807c9c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.107+0000 7fddf1234740 1 -- 172.21.15.26:0/497936896 >> 172.21.15.26:0/497936896 conn(0x560807c817b0 msgr2=0x560807c87470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- 172.21.15.26:0/497936896 shutdown_connections 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- 172.21.15.26:0/497936896 wait complete. 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- start start 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 Processor -- start 2024-04-23T18:22:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85490 0x560807c98ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c93c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81ac050 con 0x560807c85db0 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81ad4c0 con 0x560807c90480 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81ac720 con 0x560807c85490 2024-04-23T18:22:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fddf1234740 10 client.0 _ll_register_callbacks cb 0x7fddd809c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41008/0 (socket says 172.21.15.26:41008) 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.108+0000 7fdde77fe640 1 -- 172.21.15.26:0/4106000696 learned_addr learned my addr 172.21.15.26:0/4106000696 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddec9e6640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c93c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 -- 172.21.15.26:0/4106000696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85490 msgr2=0x560807c98ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85490 0x560807c98ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 -- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 msgr2=0x560807c93c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c93c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 -- 172.21.15.26:0/4106000696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560807d068f0 con 0x560807c85db0 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fdde77fe640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00ca90 tx=0x7fdddc0133a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddc6f826ecb7749 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc027030 con 0x560807c85db0 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdddc01b040 con 0x560807c85db0 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdddc015d40 con 0x560807c85db0 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdddc0137a0 con 0x560807c85db0 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddf1234740 10 client.16335 fetch_fsmap learned FSMap version 34 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560807d07af0 con 0x560807c85db0 2024-04-23T18:22:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdddc015d40 con 0x560807c85db0 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.109+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7fdddc035b60 con 0x560807c85db0 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 10 client.16335 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 10 client.16335 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 20 client.16335 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807d08240 0x560807d0a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 4 client.16335 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dd9551a1", "release": true, "await": true}] 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dd9551a1", "release": true, "await": true}) -- 0x560807bf80f0 con 0x560807d08240 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fdde7fff640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807d08240 0x560807d0a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fdde7fff640 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807d08240 0x560807d0a660 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21bcadd2cd7d9c40 server_cookie=8cc8c831f07dd068 in_seq=0 out_seq=0 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.110+0000 7fddc77fe640 10 client.16335 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddc77fe640 1 -- 172.21.15.26:0/4106000696 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560807bf80f0 con 0x560807d08240 2024-04-23T18:22:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddc77fe640 10 client.16335 handle_command_reply: tid=0 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 client.16335 shutdown 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 msgr2=0x560807c93520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00ca90 tx=0x7fdddc0133a0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 shutdown_connections 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x560807d08240 0x560807d0a660 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560807c90480 0x560807c93c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddc77fe640 0 client.16335 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x560807c85db0 0x560807c93520 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 --2- 172.21.15.26:0/4106000696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560807c85490 0x560807c98ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 >> 172.21.15.26:0/4106000696 conn(0x560807c817b0 msgr2=0x560807c9b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 shutdown_connections 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 1 -- 172.21.15.26:0/4106000696 wait complete. 2024-04-23T18:22:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.114+0000 7fddf1234740 20 client.16335 trim_cache size 0 max 16384 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 34, 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 28, 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "dd9551a1": { 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 28, 2024-04-23T18:22:32.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 81.1, 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:22:32.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:22:32.128 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 msgr2=0x7fddd00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 0x7fddd00a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009dc0 tx=0x7fdde0048000 comp rx=0 tx=0).stop 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 msgr2=0x7fdde8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c920 tx=0x7fddd800cde0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 shutdown_connections 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fddd00a3d20 0x7fddd00a61e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdde8107270 0x7fdde8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdde819cf30 0x7fdde819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 --2- 172.21.15.26:0/3492479203 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fdde806a840 0x7fdde8198670 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 >> 172.21.15.26:0/3492479203 conn(0x7fdde8077eb0 msgr2=0x7fdde80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 shutdown_connections 2024-04-23T18:22:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:22:32.115+0000 7fddee449640 1 -- 172.21.15.26:0/3492479203 wait complete. 2024-04-23T18:22:32.131 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dd9551a1', seconds elapsed: 22.2 2024-04-23T18:22:32.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 315165 5.81 MB/sec execute 147 sec latency 21658.503 ms 2024-04-23T18:22:33.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:32 smithi110 ceph-mon[32585]: pgmap v165: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 927 KiB/s wr, 51 op/s 2024-04-23T18:22:33.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:32 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4063300889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:32 smithi026 ceph-mon[29364]: pgmap v165: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 927 KiB/s wr, 51 op/s 2024-04-23T18:22:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:32 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4063300889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:33.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:32 smithi131 ceph-mon[33041]: pgmap v165: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 927 KiB/s wr, 51 op/s 2024-04-23T18:22:33.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:32 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4063300889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:33.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 315505 5.78 MB/sec execute 148 sec latency 930.427 ms 2024-04-23T18:22:34.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 315936 5.77 MB/sec execute 149 sec latency 1675.109 ms 2024-04-23T18:22:35.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:34 smithi110 ceph-mon[32585]: pgmap v166: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 313 KiB/s wr, 23 op/s 2024-04-23T18:22:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:34 smithi026 ceph-mon[29364]: pgmap v166: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 313 KiB/s wr, 23 op/s 2024-04-23T18:22:35.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:34 smithi131 ceph-mon[33041]: pgmap v166: 97 pgs: 97 active+clean; 165 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 313 KiB/s wr, 23 op/s 2024-04-23T18:22:35.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 316028 5.73 MB/sec execute 150 sec latency 862.289 ms 2024-04-23T18:22:36.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 317905 5.74 MB/sec execute 151 sec latency 864.031 ms 2024-04-23T18:22:37.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:36 smithi110 ceph-mon[32585]: pgmap v167: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 480 KiB/s wr, 50 op/s 2024-04-23T18:22:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:36 smithi026 ceph-mon[29364]: pgmap v167: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 480 KiB/s wr, 50 op/s 2024-04-23T18:22:37.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:36 smithi131 ceph-mon[33041]: pgmap v167: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 480 KiB/s wr, 50 op/s 2024-04-23T18:22:37.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319617 5.79 MB/sec execute 152 sec latency 260.497 ms 2024-04-23T18:22:38.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.75 MB/sec execute 153 sec latency 708.343 ms 2024-04-23T18:22:39.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:38 smithi110 ceph-mon[32585]: pgmap v168: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:39.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:38 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:39.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:38 smithi026 ceph-mon[29364]: pgmap v168: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:39.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:38 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:39.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:38 smithi131 ceph-mon[33041]: pgmap v168: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:39.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:38 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:39.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.71 MB/sec execute 154 sec latency 1708.460 ms 2024-04-23T18:22:40.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.67 MB/sec execute 155 sec latency 2708.539 ms 2024-04-23T18:22:41.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:40 smithi110 ceph-mon[32585]: pgmap v169: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:40 smithi026 ceph-mon[29364]: pgmap v169: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:41.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:40 smithi131 ceph-mon[33041]: pgmap v169: 97 pgs: 97 active+clean; 166 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 177 KiB/s wr, 32 op/s 2024-04-23T18:22:41.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.64 MB/sec execute 156 sec latency 3708.679 ms 2024-04-23T18:22:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 -- 172.21.15.26:0/118114336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc1021d0 msgr2=0x7f83fc1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 --2- 172.21.15.26:0/118114336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc1021d0 0x7f83fc1025b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009900 tx=0x7f83f802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 -- 172.21.15.26:0/118114336 shutdown_connections 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 --2- 172.21.15.26:0/118114336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc102b80 0x7f83fc106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 --2- 172.21.15.26:0/118114336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 0x7f83fc10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 --2- 172.21.15.26:0/118114336 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc1021d0 0x7f83fc1025b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 -- 172.21.15.26:0/118114336 >> 172.21.15.26:0/118114336 conn(0x7f83fc0793e0 msgr2=0x7f83fc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 -- 172.21.15.26:0/118114336 shutdown_connections 2024-04-23T18:22:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.009+0000 7f84041c6640 1 -- 172.21.15.26:0/118114336 wait complete. 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 Processor -- start 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 -- start start 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc1021d0 0x7f83fc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 0x7f83fc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc112260 con 0x7f83fc074cd0 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1120e0 con 0x7f83fc1021d0 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f84041c6640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1123e0 con 0x7f83fc1072a0 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f8401f62640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc1021d0 0x7f83fc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.010+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42954/0 (socket says 172.21.15.26:42954) 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 -- 172.21.15.26:0/233398219 learned_addr learned my addr 172.21.15.26:0/233398219 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8401761640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 0x7f83fc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 -- 172.21.15.26:0/233398219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 msgr2=0x7f83fc074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 0x7f83fc074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 -- 172.21.15.26:0/233398219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc1021d0 msgr2=0x7f83fc074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc1021d0 0x7f83fc074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 -- 172.21.15.26:0/233398219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc1a6f90 con 0x7f83fc074cd0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f8402763640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b9e0 tx=0x7f83f400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f83f2ffd640 1 -- 172.21.15.26:0/233398219 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4019070 con 0x7f83fc074cd0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f83f2ffd640 1 -- 172.21.15.26:0/233398219 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f4014440 con 0x7f83fc074cd0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83fc1a7280 con 0x7f83fc074cd0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f83f2ffd640 1 -- 172.21.15.26:0/233398219 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f40075d0 con 0x7f83fc074cd0 2024-04-23T18:22:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.011+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83fc1a7820 con 0x7f83fc074cd0 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.013+0000 7f83f2ffd640 1 -- 172.21.15.26:0/233398219 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f83f401c070 con 0x7f83fc074cd0 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f8406fb8740 1 Processor -- start 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f8406fb8740 1 -- start start 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f83f2ffd640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 0x7f83e80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b16b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.014+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8406fb8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56242b0c5970 con 0x56242b16b800 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8406fb8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56242b0624a0 con 0x56242b160f40 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8406fb8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56242b0c1930 con 0x56242b160620 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401761640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 0x7f83e80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f83f2ffd640 1 -- 172.21.15.26:0/233398219 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f83f40cfb80 con 0x7f83fc074cd0 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401761640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b16b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:49660/0 (socket says 172.21.15.26:49660) 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401761640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b16b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50042/0 (socket says 172.21.15.26:50042) 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42964/0 (socket says 172.21.15.26:42964) 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 -- 172.21.15.26:0/3882816477 learned_addr learned my addr 172.21.15.26:0/3882816477 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 -- 172.21.15.26:0/3882816477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 msgr2=0x56242b16b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.014 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b16b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401761640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 0x7f83e80a6330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83fc075ce0 tx=0x7f83ec0138e0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 -- 172.21.15.26:0/3882816477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 msgr2=0x56242b16dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 -- 172.21.15.26:0/3882816477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242add8210 con 0x56242b160620 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8402763640 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.015+0000 7f8401f62640 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009900 tx=0x7f83f8032e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29bfd1cdb9cf7c06 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f83e7fff640 1 -- 172.21.15.26:0/3882816477 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8040070 con 0x56242b160620 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f83e7fff640 1 -- 172.21.15.26:0/3882816477 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f80316a0 con 0x56242b160620 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f83e7fff640 1 -- 172.21.15.26:0/3882816477 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8035040 con 0x56242b160620 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 -- 172.21.15.26:0/3882816477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 msgr2=0x56242b160a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009900 tx=0x7f83f8032e60 comp rx=0 tx=0).stop 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 -- 172.21.15.26:0/3882816477 shutdown_connections 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b16b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.015 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b16dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 --2- 172.21.15.26:0/3882816477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b160a00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.016+0000 7f8406fb8740 1 -- 172.21.15.26:0/3882816477 >> 172.21.15.26:0/3882816477 conn(0x56242b15cb30 msgr2=0x56242b15cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- 172.21.15.26:0/3882816477 shutdown_connections 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- 172.21.15.26:0/3882816477 wait complete. 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- start start 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 Processor -- start 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b16b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b168400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56242b16e3d0 con 0x56242b16b800 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56242b0600b0 con 0x56242b160f40 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.017+0000 7f8406fb8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56242b16e550 con 0x56242b160620 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8406fb8740 10 client.0 _ll_register_callbacks cb 0x56242b04dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42968/0 (socket says 172.21.15.26:42968) 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 -- 172.21.15.26:0/1704434139 learned_addr learned my addr 172.21.15.26:0/1704434139 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8401f62640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b16b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8401761640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b168400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 -- 172.21.15.26:0/1704434139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 msgr2=0x56242b16b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b16b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 -- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 msgr2=0x56242b168400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b168400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 -- 172.21.15.26:0/1704434139 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242aea4620 con 0x56242b16b800 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8402763640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f83f400cfd0 tx=0x7f83f409bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276d277040bc621d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f83e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f40b5060 con 0x56242b16b800 2024-04-23T18:22:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f83f40145f0 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.018+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83f80316a0 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83f4098280 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 10 client.16347 fetch_fsmap learned FSMap version 35 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56242b17a310 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f80316a0 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f83f400caa0 con 0x56242b16b800 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 10 client.16347 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 20 client.16347 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b17b7d0 0x56242b17dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 4 client.16347 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56242b0d3390 con 0x56242b17b7d0 2024-04-23T18:22:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.019+0000 7f8401f62640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b17b7d0 0x56242b17dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.020+0000 7f83e57fa640 10 client.16347 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.020+0000 7f8401f62640 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b17b7d0 0x56242b17dc20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c080f48ca26dec10 server_cookie=5554c2aa8a1ee2e9 in_seq=0 out_seq=0 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f83e57fa640 1 -- 172.21.15.26:0/1704434139 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56242b0d3390 con 0x56242b17b7d0 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f83e57fa640 10 client.16347 handle_command_reply: tid=0 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 client.16347 shutdown 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 msgr2=0x56242b168940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f83f400cfd0 tx=0x7f83f409bcd0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 shutdown_connections 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b17b7d0 0x56242b17dc20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160f40 0x56242b168400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f83e57fa640 0 client.16347 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b16b800 0x56242b168940 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1704434139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b160620 0x56242b16b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 >> 172.21.15.26:0/1704434139 conn(0x56242b15cb30 msgr2=0x56242b162860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 shutdown_connections 2024-04-23T18:22:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 1 -- 172.21.15.26:0/1704434139 wait complete. 2024-04-23T18:22:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.021+0000 7f8406fb8740 20 client.16347 trim_cache size 0 max 16384 2024-04-23T18:22:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 Processor -- start 2024-04-23T18:22:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 -- start start 2024-04-23T18:22:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b16ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b183a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1b25e0 con 0x56242b160f40 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1afe50 con 0x56242b160620 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8406fb8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1adca0 con 0x56242b16b800 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8401761640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8401761640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42980/0 (socket says 172.21.15.26:42980) 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.036+0000 7f8401761640 1 -- 172.21.15.26:0/209961560 learned_addr learned my addr 172.21.15.26:0/209961560 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8402763640 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b183a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401f62640 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b16ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 -- 172.21.15.26:0/209961560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 msgr2=0x56242b183a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b183a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 -- 172.21.15.26:0/209961560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 msgr2=0x56242b16ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b16ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 -- 172.21.15.26:0/209961560 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242b166950 con 0x56242b160f40 2024-04-23T18:22:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f8401761640 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0118e0 tx=0x7f83ec01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2caa3bec0618b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f83e77fe640 1 -- 172.21.15.26:0/209961560 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0208d0 con 0x56242b160f40 2024-04-23T18:22:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f83e77fe640 1 -- 172.21.15.26:0/209961560 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ec02a460 con 0x56242b160f40 2024-04-23T18:22:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.037+0000 7f83e77fe640 1 -- 172.21.15.26:0/209961560 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec01f6a0 con 0x56242b160f40 2024-04-23T18:22:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- 172.21.15.26:0/209961560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 msgr2=0x56242b183530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0118e0 tx=0x7f83ec01c000 comp rx=0 tx=0).stop 2024-04-23T18:22:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- 172.21.15.26:0/209961560 shutdown_connections 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b183a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b183530 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 --2- 172.21.15.26:0/209961560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b16ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- 172.21.15.26:0/209961560 >> 172.21.15.26:0/209961560 conn(0x56242b15cb30 msgr2=0x56242b1683c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- 172.21.15.26:0/209961560 shutdown_connections 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- 172.21.15.26:0/209961560 wait complete. 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 -- start start 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.038+0000 7f8406fb8740 1 Processor -- start 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b173c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b171610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1ae310 con 0x56242b160f40 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1af780 con 0x56242b160620 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc1ae9e0 con 0x56242b16b800 2024-04-23T18:22:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 10 client.0 _ll_register_callbacks cb 0x7f83f409b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42988/0 (socket says 172.21.15.26:42988) 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 -- 172.21.15.26:0/1366483744 learned_addr learned my addr 172.21.15.26:0/1366483744 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401f62640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b173c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8402763640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b171610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 -- 172.21.15.26:0/1366483744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 msgr2=0x56242b171610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b171610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 -- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 msgr2=0x56242b173c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b173c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 -- 172.21.15.26:0/1366483744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242b171e10 con 0x56242b160f40 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8401761640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00be30 tx=0x7f83ec005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f605520ed68b33d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f83e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec02a8e0 con 0x56242b160f40 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f83ec02aa80 con 0x56242b160f40 2024-04-23T18:22:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.039+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83ec02a460 con 0x56242b160f40 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83ec0208d0 con 0x56242b160f40 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 10 client.16353 fetch_fsmap learned FSMap version 35 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56242b1e1ec0 con 0x56242b160f40 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ec02a460 con 0x56242b160f40 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f83ec0046c0 con 0x56242b160f40 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 10 client.16353 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 20 client.16353 populate_metadata read hostname 'smithi026' 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b1e25d0 0x56242b1e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 4 client.16353 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56242b0cd200 con 0x56242b1e25d0 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.040+0000 7f8401f62640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b1e25d0 0x56242b1e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.041+0000 7f83e57fa640 10 client.16353 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:42.041+0000 7f8401f62640 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b1e25d0 0x56242b1e49f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4927d3fc86ff932 server_cookie=616e6ea02c89866f in_seq=0 out_seq=0 2024-04-23T18:22:42.674 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.60 MB/sec execute 157 sec latency 4708.802 ms 2024-04-23T18:22:43.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:42 smithi110 ceph-mon[32585]: pgmap v170: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 488 KiB/s wr, 77 op/s 2024-04-23T18:22:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:42 smithi026 ceph-mon[29364]: pgmap v170: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 488 KiB/s wr, 77 op/s 2024-04-23T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:42 smithi131 ceph-mon[33041]: pgmap v170: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 488 KiB/s wr, 77 op/s 2024-04-23T18:22:43.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.57 MB/sec execute 158 sec latency 5708.955 ms 2024-04-23T18:22:44.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.53 MB/sec execute 159 sec latency 6709.064 ms 2024-04-23T18:22:45.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:44 smithi110 ceph-mon[32585]: pgmap v171: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 479 KiB/s wr, 72 op/s 2024-04-23T18:22:45.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:44 smithi026 ceph-mon[29364]: pgmap v171: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 479 KiB/s wr, 72 op/s 2024-04-23T18:22:45.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:44 smithi131 ceph-mon[33041]: pgmap v171: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 479 KiB/s wr, 72 op/s 2024-04-23T18:22:45.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.50 MB/sec execute 160 sec latency 7709.115 ms 2024-04-23T18:22:46.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.46 MB/sec execute 161 sec latency 8709.157 ms 2024-04-23T18:22:47.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:46 smithi110 ceph-mon[32585]: pgmap v172: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 588 KiB/s wr, 89 op/s 2024-04-23T18:22:47.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:46 smithi026 ceph-mon[29364]: pgmap v172: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 588 KiB/s wr, 89 op/s 2024-04-23T18:22:47.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:46 smithi131 ceph-mon[33041]: pgmap v172: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 588 KiB/s wr, 89 op/s 2024-04-23T18:22:47.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.43 MB/sec execute 162 sec latency 9709.245 ms 2024-04-23T18:22:48.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.40 MB/sec execute 163 sec latency 10709.402 ms 2024-04-23T18:22:49.192 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:48 smithi110 ceph-mon[32585]: pgmap v173: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:49.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:48 smithi026 ceph-mon[29364]: pgmap v173: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:49.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:48 smithi131 ceph-mon[33041]: pgmap v173: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:49.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.36 MB/sec execute 164 sec latency 11709.505 ms 2024-04-23T18:22:50.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 319841 5.33 MB/sec execute 165 sec latency 12709.558 ms 2024-04-23T18:22:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f83e57fa640 1 -- 172.21.15.26:0/1366483744 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56242b0cd200 con 0x56242b1e25d0 2024-04-23T18:22:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f83e57fa640 10 client.16353 handle_command_reply: tid=0 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 client.16353 shutdown 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 msgr2=0x56242b171000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00be30 tx=0x7f83ec005e50 comp rx=0 tx=0).stop 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 shutdown_connections 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56242b1e25d0 0x56242b1e49f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56242b16b800 0x56242b171610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56242b160f40 0x56242b171000 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f83e57fa640 0 client.16353 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 --2- 172.21.15.26:0/1366483744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56242b160620 0x56242b173c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 >> 172.21.15.26:0/1366483744 conn(0x56242b15cb30 msgr2=0x56242b15b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 shutdown_connections 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 1 -- 172.21.15.26:0/1366483744 wait complete. 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.708+0000 7f8406fb8740 20 client.16353 trim_cache size 0 max 16384 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:22:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 msgr2=0x7f83e80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 0x7f83e80a6330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83fc075ce0 tx=0x7f83ec0138e0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 msgr2=0x7f83fc1a6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b9e0 tx=0x7f83f400bea0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 shutdown_connections 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f83e80a3e70 0x7f83e80a6330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f83fc074cd0 0x7f83fc1a6890 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f83fc1072a0 0x7f83fc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 --2- 172.21.15.26:0/233398219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83fc1021d0 0x7f83fc074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 >> 172.21.15.26:0/233398219 conn(0x7f83fc0793e0 msgr2=0x7f83fc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 shutdown_connections 2024-04-23T18:22:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:22:50.709+0000 7f84041c6640 1 -- 172.21.15.26:0/233398219 wait complete. 2024-04-23T18:22:51.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:50 smithi110 ceph-mon[32585]: pgmap v174: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:50 smithi026 ceph-mon[29364]: pgmap v174: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:51.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:50 smithi131 ceph-mon[33041]: pgmap v174: 97 pgs: 97 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 421 KiB/s wr, 62 op/s 2024-04-23T18:22:51.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 321737 5.35 MB/sec execute 166 sec latency 12726.351 ms 2024-04-23T18:22:52.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 323471 5.38 MB/sec execute 167 sec latency 12.180 ms 2024-04-23T18:22:53.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:52 smithi110 ceph-mon[32585]: pgmap v175: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 439 KiB/s wr, 69 op/s 2024-04-23T18:22:53.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:52 smithi026 ceph-mon[29364]: pgmap v175: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 439 KiB/s wr, 69 op/s 2024-04-23T18:22:53.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:52 smithi131 ceph-mon[33041]: pgmap v175: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 439 KiB/s wr, 69 op/s 2024-04-23T18:22:53.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 325788 5.42 MB/sec execute 168 sec latency 14.745 ms 2024-04-23T18:22:54.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 327240 5.43 MB/sec execute 169 sec latency 42.806 ms 2024-04-23T18:22:55.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:54 smithi110 ceph-mon[32585]: pgmap v176: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 127 KiB/s wr, 23 op/s 2024-04-23T18:22:55.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:54 smithi026 ceph-mon[29364]: pgmap v176: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 127 KiB/s wr, 23 op/s 2024-04-23T18:22:55.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:54 smithi131 ceph-mon[33041]: pgmap v176: 97 pgs: 97 active+clean; 169 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 127 KiB/s wr, 23 op/s 2024-04-23T18:22:55.675 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328727 5.44 MB/sec execute 170 sec latency 342.840 ms 2024-04-23T18:22:56.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:55 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:56.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:55 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:56.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:55 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:56.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328727 5.41 MB/sec execute 171 sec latency 1342.980 ms 2024-04-23T18:22:57.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:56 smithi026 ceph-mon[29364]: pgmap v177: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 833 KiB/s wr, 119 op/s 2024-04-23T18:22:57.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:56 smithi131 ceph-mon[33041]: pgmap v177: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 833 KiB/s wr, 119 op/s 2024-04-23T18:22:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:56 smithi110 ceph-mon[32585]: pgmap v177: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 833 KiB/s wr, 119 op/s 2024-04-23T18:22:57.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328739 5.38 MB/sec execute 172 sec latency 1789.530 ms 2024-04-23T18:22:58.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328739 5.35 MB/sec execute 173 sec latency 1540.073 ms 2024-04-23T18:22:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:22:58 smithi026 ceph-mon[29364]: pgmap v178: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:22:59.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:22:58 smithi131 ceph-mon[33041]: pgmap v178: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:22:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:22:58 smithi110 ceph-mon[32585]: pgmap v178: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:22:59.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328739 5.31 MB/sec execute 174 sec latency 2540.180 ms 2024-04-23T18:23:00.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328739 5.28 MB/sec execute 175 sec latency 3540.255 ms 2024-04-23T18:23:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:00 smithi026 ceph-mon[29364]: pgmap v179: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:23:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:00 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:01.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:00 smithi131 ceph-mon[33041]: pgmap v179: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:23:01.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:00 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:00 smithi110 ceph-mon[32585]: pgmap v179: 97 pgs: 97 active+clean; 177 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 KiB/s wr, 101 op/s 2024-04-23T18:23:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:00 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:01.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328829 5.26 MB/sec execute 176 sec latency 3782.539 ms 2024-04-23T18:23:02.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328897 5.23 MB/sec execute 177 sec latency 1110.937 ms 2024-04-23T18:23:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:02 smithi026 ceph-mon[29364]: pgmap v180: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 KiB/s wr, 107 op/s 2024-04-23T18:23:03.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:02 smithi131 ceph-mon[33041]: pgmap v180: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 KiB/s wr, 107 op/s 2024-04-23T18:23:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:02 smithi110 ceph-mon[32585]: pgmap v180: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 KiB/s wr, 107 op/s 2024-04-23T18:23:03.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328897 5.20 MB/sec execute 178 sec latency 1529.079 ms 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 -- 172.21.15.26:0/2114589963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9401010b0 msgr2=0x7fe940101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 --2- 172.21.15.26:0/2114589963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9401010b0 0x7fe940101490 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe930009900 tx=0x7fe93002a410 comp rx=0 tx=0).stop 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 -- 172.21.15.26:0/2114589963 shutdown_connections 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 --2- 172.21.15.26:0/2114589963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe940101a60 0x7fe940105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 --2- 172.21.15.26:0/2114589963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe940106180 0x7fe94010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 --2- 172.21.15.26:0/2114589963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe9401010b0 0x7fe940101490 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 -- 172.21.15.26:0/2114589963 >> 172.21.15.26:0/2114589963 conn(0x7fe940078000 msgr2=0x7fe940078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 -- 172.21.15.26:0/2114589963 shutdown_connections 2024-04-23T18:23:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.812+0000 7fe944e6b640 1 -- 172.21.15.26:0/2114589963 wait complete. 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.813+0000 7fe944e6b640 1 Processor -- start 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.813+0000 7fe944e6b640 1 -- start start 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 0x7fe9401986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe94019cf60 0x7fe94019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe940112200 con 0x7fe940106180 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe940112080 con 0x7fe94019cf60 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe944e6b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe940112380 con 0x7fe9401010b0 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52716/0 (socket says 172.21.15.26:52716) 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 -- 172.21.15.26:0/3825691987 learned_addr learned my addr 172.21.15.26:0/3825691987 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe94019cf60 0x7fe94019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe93e59c640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 0x7fe9401986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 -- 172.21.15.26:0/3825691987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 msgr2=0x7fe9401986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 0x7fe9401986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 -- 172.21.15.26:0/3825691987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe94019cf60 msgr2=0x7fe94019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe94019cf60 0x7fe94019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 -- 172.21.15.26:0/3825691987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94019da50 con 0x7fe940106180 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe93e59c640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 0x7fe9401986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe937fff640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe92800d770 tx=0x7fe92800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.814+0000 7fe9377fe640 1 -- 172.21.15.26:0/3825691987 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928017070 con 0x7fe940106180 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.815+0000 7fe9377fe640 1 -- 172.21.15.26:0/3825691987 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe928009c30 con 0x7fe940106180 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.815+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94019dd40 con 0x7fe940106180 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.815+0000 7fe9377fe640 1 -- 172.21.15.26:0/3825691987 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92800f620 con 0x7fe940106180 2024-04-23T18:23:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.815+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9401a5580 con 0x7fe940106180 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.817+0000 7fe9377fe640 1 -- 172.21.15.26:0/3825691987 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe92801a070 con 0x7fe940106180 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.817+0000 7fe947b7d740 1 Processor -- start 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.817+0000 7fe947b7d740 1 -- start start 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2620 0x562a904b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x562a90417970 con 0x562a904bd800 2024-04-23T18:23:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562a903b44a0 con 0x562a904b2f40 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe947b7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x562a90413930 con 0x562a904b2620 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52732/0 (socket says 172.21.15.26:52732) 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 -- 172.21.15.26:0/3697665480 learned_addr learned my addr 172.21.15.26:0/3697665480 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2620 0x562a904b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe93e59c640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe9377fe640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 0x7fe92c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 0x7fe92c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 -- 172.21.15.26:0/3697665480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2620 msgr2=0x562a904b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2620 0x562a904b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 -- 172.21.15.26:0/3697665480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 msgr2=0x562a904bd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904bd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe9377fe640 1 -- 172.21.15.26:0/3825691987 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe9280d02c0 con 0x7fe940106180 2024-04-23T18:23:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe937fff640 1 -- 172.21.15.26:0/3697665480 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9012a210 con 0x562a904bd800 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.818+0000 7fe93e59c640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.819+0000 7fe937fff640 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe92809c220 tx=0x7fe928010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aba368a83bf0dca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.819+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 0x7fe92c0a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe938005fd0 tx=0x7fe938004ab0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.819+0000 7fe934ff9640 1 -- 172.21.15.26:0/3697665480 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928021070 con 0x562a904bd800 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.819+0000 7fe934ff9640 1 -- 172.21.15.26:0/3697665480 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9280a39c0 con 0x562a904bd800 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.819+0000 7fe934ff9640 1 -- 172.21.15.26:0/3697665480 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92809c290 con 0x562a904bd800 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- 172.21.15.26:0/3697665480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 msgr2=0x562a904bfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe92809c220 tx=0x7fe928010c00 comp rx=0 tx=0).stop 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- 172.21.15.26:0/3697665480 shutdown_connections 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904bfbf0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- 172.21.15.26:0/3697665480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2620 0x562a904b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- 172.21.15.26:0/3697665480 >> 172.21.15.26:0/3697665480 conn(0x562a904aeb30 msgr2=0x562a904aef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- 172.21.15.26:0/3697665480 shutdown_connections 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- 172.21.15.26:0/3697665480 wait complete. 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 -- start start 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 Processor -- start 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.820+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904ba380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904bd800 0x562a904ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x562a904c0370 con 0x562a904b2620 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562a903b20b0 con 0x562a904bd800 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x562a904c04f0 con 0x562a904b2f40 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 10 client.0 _ll_register_callbacks cb 0x562a9039fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52742/0 (socket says 172.21.15.26:52742) 2024-04-23T18:23:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 -- 172.21.15.26:0/2781350042 learned_addr learned my addr 172.21.15.26:0/2781350042 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe937fff640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904ba380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904bd800 0x562a904ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 -- 172.21.15.26:0/2781350042 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 msgr2=0x562a904ba380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904ba380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 -- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904bd800 msgr2=0x562a904ba8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904bd800 0x562a904ba8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 -- 172.21.15.26:0/2781350042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a901f6620 con 0x562a904b2620 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe930009900 tx=0x7fe930031950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558baf63b5223d41 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930035040 con 0x562a904b2620 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe9300040c0 con 0x562a904b2620 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.821+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a90425390 con 0x562a904b2620 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe9300042c0 con 0x562a904b2620 2024-04-23T18:23:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 10 client.16365 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562a904cc200 con 0x562a904b2620 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562a90425390 con 0x562a904b2620 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe930004b00 con 0x562a904b2620 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 10 client.16365 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 20 client.16365 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a904cd6c0 0x562a904cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 4 client.16365 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a9041f200 con 0x562a904cd6c0 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.822+0000 7fe937fff640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a904cd6c0 0x562a904cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.823+0000 7fe925ffb640 10 client.16365 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.823+0000 7fe937fff640 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a904cd6c0 0x562a904cfad0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5249009724a3c6d server_cookie=d8b91825dcdfdbc7 in_seq=0 out_seq=0 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.823+0000 7fe925ffb640 1 -- 172.21.15.26:0/2781350042 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a9041f200 con 0x562a904cd6c0 2024-04-23T18:23:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.823+0000 7fe925ffb640 10 client.16365 handle_command_reply: tid=0 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 client.16365 shutdown 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 msgr2=0x562a904bd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe930009900 tx=0x7fe930031950 comp rx=0 tx=0).stop 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 shutdown_connections 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a904cd6c0 0x562a904cfad0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904bd800 0x562a904ba8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904ba380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe925ffb640 0 client.16365 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2781350042 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904bd090 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 >> 172.21.15.26:0/2781350042 conn(0x562a904aeb30 msgr2=0x562a904b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 shutdown_connections 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 1 -- 172.21.15.26:0/2781350042 wait complete. 2024-04-23T18:23:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.824+0000 7fe947b7d740 20 client.16365 trim_cache size 0 max 16384 2024-04-23T18:23:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 Processor -- start 2024-04-23T18:23:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 -- start start 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2620 0x562a904c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904d53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401b03f0 con 0x562a904bd800 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401adc00 con 0x562a904b2620 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe947b7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401aba70 con 0x562a904b2f40 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe93ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe93ed9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52750/0 (socket says 172.21.15.26:52750) 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.839+0000 7fe93ed9d640 1 -- 172.21.15.26:0/2815180676 learned_addr learned my addr 172.21.15.26:0/2815180676 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2620 0x562a904c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe937fff640 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904d53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 -- 172.21.15.26:0/2815180676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 msgr2=0x562a904d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904d53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 -- 172.21.15.26:0/2815180676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2620 msgr2=0x562a904c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2620 0x562a904c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 -- 172.21.15.26:0/2815180676 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a904b88d0 con 0x562a904bd800 2024-04-23T18:23:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe938012b50 tx=0x7fe938009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38203e1fb91d121f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe927fff640 1 -- 172.21.15.26:0/2815180676 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93801a070 con 0x562a904bd800 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe927fff640 1 -- 172.21.15.26:0/2815180676 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe938021400 con 0x562a904bd800 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.840+0000 7fe927fff640 1 -- 172.21.15.26:0/2815180676 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938013660 con 0x562a904bd800 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- 172.21.15.26:0/2815180676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 msgr2=0x562a904d5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe938012b50 tx=0x7fe938009ab0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- 172.21.15.26:0/2815180676 shutdown_connections 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904bd800 0x562a904d5920 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904b2f40 0x562a904d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2815180676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2620 0x562a904c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- 172.21.15.26:0/2815180676 >> 172.21.15.26:0/2815180676 conn(0x562a904aeb30 msgr2=0x562a904b9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- 172.21.15.26:0/2815180676 shutdown_connections 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- 172.21.15.26:0/2815180676 wait complete. 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- start start 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 Processor -- start 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 0x562a904c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401ac0c0 con 0x562a904b2620 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401ad530 con 0x562a904b2f40 2024-04-23T18:23:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401ac790 con 0x562a904bd800 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe947b7d740 10 client.0 _ll_register_callbacks cb 0x7fe9300315e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.841+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52754/0 (socket says 172.21.15.26:52754) 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 -- 172.21.15.26:0/2805698432 learned_addr learned my addr 172.21.15.26:0/2805698432 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe937fff640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 -- 172.21.15.26:0/2805698432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 msgr2=0x562a904c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 0x562a904c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 0x562a904c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 -- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 msgr2=0x562a904c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 -- 172.21.15.26:0/2805698432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a904c3a40 con 0x562a904b2620 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe937fff640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93ed9d640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 0x562a904c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe93e59c640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe930033000 tx=0x7fe9300092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dcf8e8e5408d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930035040 con 0x562a904b2620 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe9300038e0 con 0x562a904b2620 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe938021400 con 0x562a904b2620 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe930003ae0 con 0x562a904b2620 2024-04-23T18:23:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe947b7d740 10 client.16371 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.842+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562a90533ba0 con 0x562a904b2620 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe938021400 con 0x562a904b2620 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe930005060 con 0x562a904b2620 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 10 client.16371 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 20 client.16371 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a905343d0 0x562a905367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 4 client.16371 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a9039b5f0 con 0x562a905343d0 2024-04-23T18:23:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe937fff640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a905343d0 0x562a905367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe925ffb640 10 client.16371 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.843+0000 7fe937fff640 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a905343d0 0x562a905367f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d48888fd4ff5c8 server_cookie=8c9d547ba7b7f9fa in_seq=0 out_seq=0 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe925ffb640 1 -- 172.21.15.26:0/2805698432 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a9039b5f0 con 0x562a905343d0 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe925ffb640 10 client.16371 handle_command_reply: tid=0 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 client.16371 shutdown 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 msgr2=0x562a904c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe930033000 tx=0x7fe9300092f0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 shutdown_connections 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x562a905343d0 0x562a905367f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a904b2f40 0x562a904c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe925ffb640 0 client.16371 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:03.873 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562a904bd800 0x562a904c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 --2- 172.21.15.26:0/2805698432 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x562a904b2620 0x562a904c5a30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 >> 172.21.15.26:0/2805698432 conn(0x562a904aeb30 msgr2=0x562a904ba470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 shutdown_connections 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 1 -- 172.21.15.26:0/2805698432 wait complete. 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.879+0000 7fe947b7d740 20 client.16371 trim_cache size 0 max 16384 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:23:03.874 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:23:03.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 msgr2=0x7fe92c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 0x7fe92c0a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe938005fd0 tx=0x7fe938004ab0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 msgr2=0x7fe940198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:03.875 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe92800d770 tx=0x7fe92800dc30 comp rx=0 tx=0).stop 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 shutdown_connections 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe92c0a3e40 0x7fe92c0a6300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe94019cf60 0x7fe94019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe940106180 0x7fe940198be0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 --2- 172.21.15.26:0/3825691987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9401010b0 0x7fe9401986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 >> 172.21.15.26:0/3825691987 conn(0x7fe940078000 msgr2=0x7fe9400fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 shutdown_connections 2024-04-23T18:23:03.876 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:03.880+0000 7fe944e6b640 1 -- 172.21.15.26:0/3825691987 wait complete. 2024-04-23T18:23:04.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328934 5.17 MB/sec execute 179 sec latency 2271.089 ms 2024-04-23T18:23:05.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:04 smithi131 ceph-mon[33041]: pgmap v181: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-23T18:23:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:04 smithi110 ceph-mon[32585]: pgmap v181: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-23T18:23:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:04 smithi026 ceph-mon[29364]: pgmap v181: 97 pgs: 97 active+clean; 176 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-23T18:23:05.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 328934 5.14 MB/sec execute 180 sec latency 1228.866 ms 2024-04-23T18:23:06.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 330293 5.18 MB/sec execute 181 sec latency 1250.249 ms 2024-04-23T18:23:07.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:06 smithi131 ceph-mon[33041]: pgmap v182: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 998 KiB/s wr, 147 op/s 2024-04-23T18:23:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:06 smithi110 ceph-mon[32585]: pgmap v182: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 998 KiB/s wr, 147 op/s 2024-04-23T18:23:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:06 smithi026 ceph-mon[29364]: pgmap v182: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 998 KiB/s wr, 147 op/s 2024-04-23T18:23:07.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 332192 5.19 MB/sec execute 182 sec latency 19.384 ms 2024-04-23T18:23:08.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 333878 5.22 MB/sec execute 183 sec latency 13.161 ms 2024-04-23T18:23:09.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:08 smithi131 ceph-mon[33041]: pgmap v183: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:08 smithi110 ceph-mon[32585]: pgmap v183: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:08 smithi026 ceph-mon[29364]: pgmap v183: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:09.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 335823 5.23 MB/sec execute 184 sec latency 18.951 ms 2024-04-23T18:23:10.676 INFO:tasks.workunit.client.0.smithi026.stdout: 1 337479 5.27 MB/sec execute 185 sec latency 11.296 ms 2024-04-23T18:23:11.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:10 smithi131 ceph-mon[33041]: pgmap v184: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:10 smithi110 ceph-mon[32585]: pgmap v184: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:10 smithi026 ceph-mon[29364]: pgmap v184: 97 pgs: 97 active+clean; 173 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 292 KiB/s wr, 51 op/s 2024-04-23T18:23:11.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 339463 5.28 MB/sec execute 186 sec latency 22.930 ms 2024-04-23T18:23:12.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 341111 5.32 MB/sec execute 187 sec latency 12.121 ms 2024-04-23T18:23:13.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:12 smithi131 ceph-mon[33041]: pgmap v185: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-04-23T18:23:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:12 smithi110 ceph-mon[32585]: pgmap v185: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-04-23T18:23:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:12 smithi026 ceph-mon[29364]: pgmap v185: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-04-23T18:23:13.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 343101 5.33 MB/sec execute 188 sec latency 21.532 ms 2024-04-23T18:23:14.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 344751 5.36 MB/sec execute 189 sec latency 17.455 ms 2024-04-23T18:23:15.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:14 smithi131 ceph-mon[33041]: pgmap v186: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-23T18:23:15.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:14 smithi110 ceph-mon[32585]: pgmap v186: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-23T18:23:15.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:14 smithi026 ceph-mon[29364]: pgmap v186: 97 pgs: 97 active+clean; 180 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-23T18:23:15.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 346648 5.38 MB/sec execute 190 sec latency 12.745 ms 2024-04-23T18:23:16.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 348296 5.41 MB/sec execute 191 sec latency 26.637 ms 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1678605033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a681021c0 msgr2=0x7f8a681025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1678605033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a681021c0 0x7f8a681025a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009900 tx=0x7f8a6002a410 comp rx=0 tx=0).stop 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1678605033 shutdown_connections 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1678605033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a68102b70 0x7f8a68106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1678605033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a68107290 0x7f8a6810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1678605033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a681021c0 0x7f8a681025a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1678605033 >> 172.21.15.26:0/1678605033 conn(0x7f8a680fda20 msgr2=0x7f8a680ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1678605033 shutdown_connections 2024-04-23T18:23:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.972+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1678605033 wait complete. 2024-04-23T18:23:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 Processor -- start 2024-04-23T18:23:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 -- start start 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a681021c0 0x7f8a681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a6819cf60 0x7f8a6819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681121d0 con 0x7f8a68107290 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a68112050 con 0x7f8a681021c0 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.973+0000 7f8a6d1f7640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a68112350 con 0x7f8a6819cf60 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45538/0 (socket says 172.21.15.26:45538) 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 -- 172.21.15.26:0/1092923423 learned_addr learned my addr 172.21.15.26:0/1092923423 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6759e640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a6819cf60 0x7f8a6819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a681021c0 0x7f8a681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 -- 172.21.15.26:0/1092923423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a6819cf60 msgr2=0x7f8a6819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a6819cf60 0x7f8a6819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 -- 172.21.15.26:0/1092923423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a681021c0 msgr2=0x7f8a681986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a681021c0 0x7f8a681986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 -- 172.21.15.26:0/1092923423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819da50 con 0x7f8a68107290 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6659c640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00d770 tx=0x7f8a5c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a4ffff640 1 -- 172.21.15.26:0/1092923423 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c017070 con 0x7f8a68107290 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819dd40 con 0x7f8a68107290 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a4ffff640 1 -- 172.21.15.26:0/1092923423 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a5c009c30 con 0x7f8a68107290 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a4ffff640 1 -- 172.21.15.26:0/1092923423 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c00f620 con 0x7f8a68107290 2024-04-23T18:23:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.974+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a681a5580 con 0x7f8a68107290 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a4ffff640 1 -- 172.21.15.26:0/1092923423 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a5c01a070 con 0x7f8a68107290 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 Processor -- start 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 -- start start 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b65800 0x55dc67b67bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55dc67abf970 con 0x55dc67b65800 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc67a5c4a0 con 0x55dc67b5a620 2024-04-23T18:23:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.977+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55dc67abb930 con 0x55dc67b5af40 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a4ffff640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 0x7f8a500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:42456/0 (socket says 172.21.15.26:42456) 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 -- 172.21.15.26:0/763834098 learned_addr learned my addr 172.21.15.26:0/763834098 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a66d9d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6659c640 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b65800 0x55dc67b67bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 0x7f8a500a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a4ffff640 1 -- 172.21.15.26:0/1092923423 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a5c0d0290 con 0x7f8a68107290 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 -- 172.21.15.26:0/763834098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 msgr2=0x55dc67b652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 -- 172.21.15.26:0/763834098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b65800 msgr2=0x55dc67b67bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b65800 0x55dc67b67bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 -- 172.21.15.26:0/763834098 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc677d2210 con 0x55dc67b5a620 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a58014a60 tx=0x7f8a58014f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=100cdf2b39ebd58b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.978+0000 7f8a6759e640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 0x7f8a500a6280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a5801c000 tx=0x7f8a5801e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.979+0000 7f8a4d7fa640 1 -- 172.21.15.26:0/763834098 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58012e10 con 0x55dc67b5a620 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.979+0000 7f8a4d7fa640 1 -- 172.21.15.26:0/763834098 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a5800ebc0 con 0x55dc67b5a620 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.979+0000 7f8a4d7fa640 1 -- 172.21.15.26:0/763834098 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58005770 con 0x55dc67b5a620 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/763834098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 msgr2=0x55dc67b5aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a58014a60 tx=0x7f8a58014f20 comp rx=0 tx=0).stop 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/763834098 shutdown_connections 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b65800 0x55dc67b67bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/763834098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b5aa00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/763834098 >> 172.21.15.26:0/763834098 conn(0x55dc67b56b30 msgr2=0x55dc67b56f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/763834098 shutdown_connections 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/763834098 wait complete. 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.980+0000 7f8a6ffe6740 1 -- start start 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 Processor -- start 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b62ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55dc67b683d0 con 0x55dc67b5af40 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc67a5a0b0 con 0x55dc67b5a620 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55dc67b68550 con 0x55dc67b65800 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6ffe6740 10 client.0 _ll_register_callbacks cb 0x55dc67a47d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45552/0 (socket says 172.21.15.26:45552) 2024-04-23T18:23:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 -- 172.21.15.26:0/358177439 learned_addr learned my addr 172.21.15.26:0/358177439 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6759e640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b62ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 -- 172.21.15.26:0/358177439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 msgr2=0x55dc67b62ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b62ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 -- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 msgr2=0x55dc67b651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 -- 172.21.15.26:0/358177439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc6789e620 con 0x55dc67b5af40 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a6659c640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010bb0 tx=0x7f8a5c010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ddf45e04357acd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.981+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c021040 con 0x55dc67b5af40 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a5c0a0150 con 0x55dc67b5af40 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5800ebc0 con 0x55dc67b5af40 2024-04-23T18:23:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a5c0a0610 con 0x55dc67b5af40 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 10 client.16383 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dc67b74430 con 0x55dc67b5af40 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a5800ebc0 con 0x55dc67b5af40 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8a5c098180 con 0x55dc67b5af40 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 10 client.16383 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 20 client.16383 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67b75bc0 0x55dc67b77fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 4 client.16383 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc67acd390 con 0x55dc67b75bc0 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.982+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67b75bc0 0x55dc67b77fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.983+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67b75bc0 0x55dc67b77fb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eebb4be95a003503 server_cookie=d72ab9d10f73169c in_seq=0 out_seq=0 2024-04-23T18:23:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.983+0000 7f8a4aff5640 10 client.16383 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a4aff5640 1 -- 172.21.15.26:0/358177439 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc67acd390 con 0x55dc67b75bc0 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a4aff5640 10 client.16383 handle_command_reply: tid=0 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 client.16383 shutdown 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 msgr2=0x55dc67b624d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010bb0 tx=0x7f8a5c010c00 comp rx=0 tx=0).stop 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 shutdown_connections 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67b75bc0 0x55dc67b77fb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b62ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b624d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a4aff5640 0 client.16383 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/358177439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 >> 172.21.15.26:0/358177439 conn(0x55dc67b56b30 msgr2=0x55dc67b5c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 shutdown_connections 2024-04-23T18:23:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.984+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/358177439 wait complete. 2024-04-23T18:23:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:16.985+0000 7f8a6ffe6740 20 client.16383 trim_cache size 0 max 16384 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 Processor -- start 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 -- start start 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b64d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 0x55dc67b7d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681b0320 con 0x55dc67b5a620 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681adb30 con 0x55dc67b65800 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681ab9a0 con 0x55dc67b5af40 2024-04-23T18:23:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45560/0 (socket says 172.21.15.26:45560) 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 -- 172.21.15.26:0/3002480560 learned_addr learned my addr 172.21.15.26:0/3002480560 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6759e640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 0x55dc67b7d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6659c640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b64d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 -- 172.21.15.26:0/3002480560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 msgr2=0x55dc67b64d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b64d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 -- 172.21.15.26:0/3002480560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 msgr2=0x55dc67b7d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 0x55dc67b7d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a66d9d640 1 -- 172.21.15.26:0/3002480560 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc67b60a60 con 0x55dc67b5a620 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.005+0000 7f8a6759e640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 0x55dc67b7d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.006+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a6002a930 tx=0x7f8a60005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c59b76c7af3da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.006+0000 7f8a4cff9640 1 -- 172.21.15.26:0/3002480560 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6004b070 con 0x55dc67b5a620 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.006+0000 7f8a4cff9640 1 -- 172.21.15.26:0/3002480560 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a60046dc0 con 0x55dc67b5a620 2024-04-23T18:23:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.006+0000 7f8a4cff9640 1 -- 172.21.15.26:0/3002480560 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60035040 con 0x55dc67b5a620 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3002480560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 msgr2=0x55dc67b71290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a6002a930 tx=0x7f8a60005c00 comp rx=0 tx=0).stop 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3002480560 shutdown_connections 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b65800 0x55dc67b7d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b5af40 0x55dc67b64d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3002480560 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5a620 0x55dc67b71290 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3002480560 >> 172.21.15.26:0/3002480560 conn(0x55dc67b56b30 msgr2=0x55dc67b5c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3002480560 shutdown_connections 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3002480560 wait complete. 2024-04-23T18:23:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- start start 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 Processor -- start 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b6d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b6a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681abff0 con 0x55dc67b5af40 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681ad460 con 0x55dc67b5a620 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681ac6c0 con 0x55dc67b65800 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6ffe6740 10 client.0 _ll_register_callbacks cb 0x7f8a5c09bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.007+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a66d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b6d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6759e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b6a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:45572/0 (socket says 172.21.15.26:45572) 2024-04-23T18:23:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 -- 172.21.15.26:0/3508744636 learned_addr learned my addr 172.21.15.26:0/3508744636 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 -- 172.21.15.26:0/3508744636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 msgr2=0x55dc67b6a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b6a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 -- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 msgr2=0x55dc67b6d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b6d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 -- 172.21.15.26:0/3508744636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc67b6afe0 con 0x55dc67b5af40 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6659c640 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c098770 tx=0x7f8a5c098b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c290655824267d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c014030 con 0x55dc67b5af40 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a5c0bcdb0 con 0x55dc67b5af40 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a60046dc0 con 0x55dc67b5af40 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a5c021040 con 0x55dc67b5af40 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6ffe6740 10 client.16389 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.008+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dc67bdc9d0 con 0x55dc67b5af40 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a60046dc0 con 0x55dc67b5af40 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8a5c0a01e0 con 0x55dc67b5af40 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 10 client.16389 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 20 client.16389 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67bdd180 0x55dc67bdf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 4 client.16389 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dc67ac7200 con 0x55dc67bdd180 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67bdd180 0x55dc67bdf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a4aff5640 10 client.16389 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.009+0000 7f8a66d9d640 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67bdd180 0x55dc67bdf5a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9d233f0275a955a server_cookie=52a21b68d6e402b6 in_seq=0 out_seq=0 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a4aff5640 1 -- 172.21.15.26:0/3508744636 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dc67ac7200 con 0x55dc67bdd180 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a4aff5640 10 client.16389 handle_command_reply: tid=0 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 client.16389 shutdown 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 msgr2=0x55dc67b6dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c098770 tx=0x7f8a5c098b90 comp rx=0 tx=0).stop 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 shutdown_connections 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dc67bdd180 0x55dc67bdf5a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dc67b65800 0x55dc67b6a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dc67b5af40 0x55dc67b6dce0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 --2- 172.21.15.26:0/3508744636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc67b5a620 0x55dc67b6d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a4aff5640 0 client.16389 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 >> 172.21.15.26:0/3508744636 conn(0x55dc67b56b30 msgr2=0x55dc67b55420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 shutdown_connections 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.040+0000 7f8a6ffe6740 1 -- 172.21.15.26:0/3508744636 wait complete. 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6ffe6740 20 client.16389 trim_cache size 0 max 16384 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:23:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 msgr2=0x7f8a500a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 0x7f8a500a6280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a5801c000 tx=0x7f8a5801e000 comp rx=0 tx=0).stop 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 msgr2=0x7f8a68198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00d770 tx=0x7f8a5c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 shutdown_connections 2024-04-23T18:23:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a500a3dc0 0x7f8a500a6280 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a6819cf60 0x7f8a6819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a68107290 0x7f8a68198c30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 --2- 172.21.15.26:0/1092923423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a681021c0 0x7f8a681986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 >> 172.21.15.26:0/1092923423 conn(0x7f8a680fda20 msgr2=0x7f8a680ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 shutdown_connections 2024-04-23T18:23:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:17.041+0000 7f8a6d1f7640 1 -- 172.21.15.26:0/1092923423 wait complete. 2024-04-23T18:23:17.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:16 smithi131 ceph-mon[33041]: pgmap v187: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-04-23T18:23:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:16 smithi110 ceph-mon[32585]: pgmap v187: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-04-23T18:23:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:16 smithi026 ceph-mon[29364]: pgmap v187: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 262 op/s 2024-04-23T18:23:17.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 350113 5.42 MB/sec execute 192 sec latency 21.556 ms 2024-04-23T18:23:18.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 351765 5.45 MB/sec execute 193 sec latency 28.841 ms 2024-04-23T18:23:19.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:18 smithi131 ceph-mon[33041]: pgmap v188: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:19.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:18 smithi110 ceph-mon[32585]: pgmap v188: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:18 smithi026 ceph-mon[29364]: pgmap v188: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:19.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 353614 5.46 MB/sec execute 194 sec latency 14.965 ms 2024-04-23T18:23:20.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:19 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:20.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:19 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:20.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:19 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:20.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 355261 5.49 MB/sec execute 195 sec latency 17.191 ms 2024-04-23T18:23:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:20 smithi131 ceph-mon[33041]: pgmap v189: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:20 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:20 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:21.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:20 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:23:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:20 smithi110 ceph-mon[32585]: pgmap v189: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:20 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:20 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:20 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:23:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:20 smithi026 ceph-mon[29364]: pgmap v189: 97 pgs: 97 active+clean; 187 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-23T18:23:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:20 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:20 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:20 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:23:21.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 357151 5.50 MB/sec execute 196 sec latency 15.540 ms 2024-04-23T18:23:22.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 358858 5.53 MB/sec execute 197 sec latency 21.835 ms 2024-04-23T18:23:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:22 smithi131 ceph-mon[33041]: pgmap v190: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-04-23T18:23:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:22 smithi110 ceph-mon[32585]: pgmap v190: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-04-23T18:23:23.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:22 smithi026 ceph-mon[29364]: pgmap v190: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-04-23T18:23:23.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 360831 5.54 MB/sec execute 198 sec latency 17.815 ms 2024-04-23T18:23:24.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 362384 5.57 MB/sec execute 199 sec latency 31.478 ms 2024-04-23T18:23:25.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:24 smithi131 ceph-mon[33041]: pgmap v191: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-04-23T18:23:25.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:24 smithi110 ceph-mon[32585]: pgmap v191: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-04-23T18:23:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:24 smithi026 ceph-mon[29364]: pgmap v191: 97 pgs: 97 active+clean; 194 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-04-23T18:23:25.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 364367 5.58 MB/sec execute 200 sec latency 19.140 ms 2024-04-23T18:23:26.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 366016 5.61 MB/sec execute 201 sec latency 12.251 ms 2024-04-23T18:23:27.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:26 smithi131 ceph-mon[33041]: pgmap v192: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-04-23T18:23:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:26 smithi110 ceph-mon[32585]: pgmap v192: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-04-23T18:23:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:26 smithi026 ceph-mon[29364]: pgmap v192: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-04-23T18:23:27.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 367830 5.62 MB/sec execute 202 sec latency 14.098 ms 2024-04-23T18:23:28.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 369606 5.65 MB/sec execute 203 sec latency 17.805 ms 2024-04-23T18:23:29.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:28 smithi131 ceph-mon[33041]: pgmap v193: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:29.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:28 smithi110 ceph-mon[32585]: pgmap v193: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:29.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:28 smithi026 ceph-mon[29364]: pgmap v193: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:29.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 371661 5.68 MB/sec execute 204 sec latency 14.124 ms 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 -- 172.21.15.26:0/1193581640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac103b00 msgr2=0x7f65ac10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 --2- 172.21.15.26:0/1193581640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac103b00 0x7f65ac10bfe0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009900 tx=0x7f65a002a480 comp rx=0 tx=0).stop 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 -- 172.21.15.26:0/1193581640 shutdown_connections 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 --2- 172.21.15.26:0/1193581640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac10c5b0 0x7f65ac10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 --2- 172.21.15.26:0/1193581640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac103b00 0x7f65ac10bfe0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 --2- 172.21.15.26:0/1193581640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 0x7f65ac1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.136+0000 7f65b137d640 1 -- 172.21.15.26:0/1193581640 >> 172.21.15.26:0/1193581640 conn(0x7f65ac077f60 msgr2=0x7f65ac078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 -- 172.21.15.26:0/1193581640 shutdown_connections 2024-04-23T18:23:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 -- 172.21.15.26:0/1193581640 wait complete. 2024-04-23T18:23:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 Processor -- start 2024-04-23T18:23:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 -- start start 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 0x7f65ac19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.137+0000 7f65b137d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 0x7f65ac19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65b137d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 0x7f65ac198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65b137d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1121f0 con 0x7f65ac10c5b0 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65b137d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac112070 con 0x7f65ac103b00 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65b137d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac112370 con 0x7f65ac1031e0 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 0x7f65ac198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 0x7f65ac19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aaffd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 0x7f65ac19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 0x7f65ac198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47658/0 (socket says 172.21.15.26:47658) 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65ab7fe640 1 -- 172.21.15.26:0/1938470864 learned_addr learned my addr 172.21.15.26:0/1938470864 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1938470864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 msgr2=0x7f65ac19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 0x7f65ac19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 msgr2=0x7f65ac198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 0x7f65ac198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1938470864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac198d10 con 0x7f65ac103b00 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 0x7f65ac19e920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f65a0006010 tx=0x7f65a0004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.138+0000 7f6593fff640 1 -- 172.21.15.26:0/1938470864 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a003f070 con 0x7f65ac103b00 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.139+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac198fa0 con 0x7f65ac103b00 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.139+0000 7f6593fff640 1 -- 172.21.15.26:0/1938470864 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a0003ca0 con 0x7f65ac103b00 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.139+0000 7f6593fff640 1 -- 172.21.15.26:0/1938470864 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0035070 con 0x7f65ac103b00 2024-04-23T18:23:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.139+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac1a52a0 con 0x7f65ac103b00 2024-04-23T18:23:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f6593fff640 1 -- 172.21.15.26:0/1938470864 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65a003b460 con 0x7f65ac103b00 2024-04-23T18:23:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 Processor -- start 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 -- start start 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f6593fff640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 0x7f65940a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207ac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ef20711970 con 0x55ef207ace40 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef206ae4a0 con 0x55ef207bbc10 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.141+0000 7f65b4173740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ef2070d930 con 0x55ef207ac520 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 0x7f65940a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f6593fff640 1 -- 172.21.15.26:0/1938470864 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f65a00f0bc0 con 0x7f65ac103b00 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207be000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:49046/0 (socket says 172.21.15.26:49046) 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47666/0 (socket says 172.21.15.26:47666) 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65ab7fe640 1 -- 172.21.15.26:0/1825041116 learned_addr learned my addr 172.21.15.26:0/1825041116 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aaffd640 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1825041116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 msgr2=0x55ef207ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1825041116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 msgr2=0x55ef207be000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207be000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 -- 172.21.15.26:0/1825041116 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef20424210 con 0x55ef207ace40 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 0x7f65940a6280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009fd0 tx=0x7f65a4013460 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f65a00ee340 tx=0x7f65a00bc5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce45bf965136c914 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65917fa640 1 -- 172.21.15.26:0/1825041116 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0048040 con 0x55ef207ace40 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65917fa640 1 -- 172.21.15.26:0/1825041116 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a00a7040 con 0x55ef207ace40 2024-04-23T18:23:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.142+0000 7f65917fa640 1 -- 172.21.15.26:0/1825041116 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a00dd200 con 0x55ef207ace40 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- 172.21.15.26:0/1825041116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 msgr2=0x55ef207bb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f65a00ee340 tx=0x7f65a00bc5a0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- 172.21.15.26:0/1825041116 shutdown_connections 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207bb6d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 --2- 172.21.15.26:0/1825041116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- 172.21.15.26:0/1825041116 >> 172.21.15.26:0/1825041116 conn(0x55ef207a8b30 msgr2=0x55ef207a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- 172.21.15.26:0/1825041116 shutdown_connections 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- 172.21.15.26:0/1825041116 wait complete. 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.143+0000 7f65b4173740 1 -- start start 2024-04-23T18:23:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 Processor -- start 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ac520 0x55ef207bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 0x55ef207b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55ef20711970 con 0x55ef207bbc10 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef206ac0b0 con 0x55ef207ac520 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55ef207be600 con 0x55ef207ace40 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65b4173740 10 client.0 _ll_register_callbacks cb 0x55ef20699d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65aaffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ac520 0x55ef207bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47682/0 (socket says 172.21.15.26:47682) 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 -- 172.21.15.26:0/4071389562 learned_addr learned my addr 172.21.15.26:0/4071389562 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 0x55ef207b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 -- 172.21.15.26:0/4071389562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 msgr2=0x55ef207b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 0x55ef207b87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 -- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ac520 msgr2=0x55ef207bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ac520 0x55ef207bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 -- 172.21.15.26:0/4071389562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef204f0620 con 0x55ef207bbc10 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 0x55ef207b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009d00 tx=0x7f65a4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123dc39560f05e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.144+0000 7f658eff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4025070 con 0x55ef207bbc10 2024-04-23T18:23:30.146 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a00a7040 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f65a4009040 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f65a402a460 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 10 client.16401 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ef207ca3a0 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a00a7040 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f65a4020070 con 0x55ef207bbc10 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 10 client.16401 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 20 client.16401 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef207cb900 0x55ef207cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 4 client.16401 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef2071f390 con 0x55ef207cb900 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.145+0000 7f65aaffd640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef207cb900 0x55ef207cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.146+0000 7f65aaffd640 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef207cb900 0x55ef207cdd10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6d6ccbd44dff69c server_cookie=edc24995a5d48bbb in_seq=0 out_seq=0 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.146+0000 7f658eff5640 10 client.16401 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.146+0000 7f658eff5640 1 -- 172.21.15.26:0/4071389562 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef2071f390 con 0x55ef207cb900 2024-04-23T18:23:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.146+0000 7f658eff5640 10 client.16401 handle_command_reply: tid=0 2024-04-23T18:23:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.146+0000 7f65b4173740 1 client.16401 shutdown 2024-04-23T18:23:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 msgr2=0x55ef207b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009d00 tx=0x7f65a4009500 comp rx=0 tx=0).stop 2024-04-23T18:23:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 shutdown_connections 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef207cb900 0x55ef207cdd10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207bbc10 0x55ef207b8d20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ace40 0x55ef207b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 --2- 172.21.15.26:0/4071389562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ac520 0x55ef207bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f658eff5640 0 client.16401 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 >> 172.21.15.26:0/4071389562 conn(0x55ef207a8b30 msgr2=0x55ef207ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 shutdown_connections 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 1 -- 172.21.15.26:0/4071389562 wait complete. 2024-04-23T18:23:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.147+0000 7f65b4173740 20 client.16401 trim_cache size 0 max 16384 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 Processor -- start 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 -- start start 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ace40 0x55ef207bb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 0x55ef207d3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1ae250 con 0x55ef207ac520 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1aba60 con 0x55ef207ace40 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65b4173740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1a98b0 con 0x55ef207bbc10 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47688/0 (socket says 172.21.15.26:47688) 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 -- 172.21.15.26:0/3670567848 learned_addr learned my addr 172.21.15.26:0/3670567848 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ace40 0x55ef207bb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 0x55ef207d3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 -- 172.21.15.26:0/3670567848 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 msgr2=0x55ef207d3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 0x55ef207d3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 -- 172.21.15.26:0/3670567848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ace40 msgr2=0x55ef207bb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ace40 0x55ef207bb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65aaffd640 1 -- 172.21.15.26:0/3670567848 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef207ce390 con 0x55ef207ac520 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.163+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 0x55ef207d3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.164+0000 7f65aaffd640 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f659c00bc00 tx=0x7f659c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cc3b172d488392 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.164+0000 7f6590ff9640 1 -- 172.21.15.26:0/3670567848 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c0201b0 con 0x55ef207ac520 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.164+0000 7f6590ff9640 1 -- 172.21.15.26:0/3670567848 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f659c026070 con 0x55ef207ac520 2024-04-23T18:23:30.158 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.164+0000 7f6590ff9640 1 -- 172.21.15.26:0/3670567848 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c011af0 con 0x55ef207ac520 2024-04-23T18:23:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.164+0000 7f65b4173740 1 -- 172.21.15.26:0/3670567848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 msgr2=0x55ef207c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f659c00bc00 tx=0x7f659c00b390 comp rx=0 tx=0).stop 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- 172.21.15.26:0/3670567848 shutdown_connections 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207ace40 0x55ef207bb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207bbc10 0x55ef207d3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- 172.21.15.26:0/3670567848 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ac520 0x55ef207c7340 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- 172.21.15.26:0/3670567848 >> 172.21.15.26:0/3670567848 conn(0x55ef207a8b30 msgr2=0x55ef207b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- 172.21.15.26:0/3670567848 shutdown_connections 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- 172.21.15.26:0/3670567848 wait complete. 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- start start 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 Processor -- start 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1a9f20 con 0x55ef207ace40 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1ab390 con 0x55ef207bbc10 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac1aa5f0 con 0x55ef207ac520 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65b4173740 10 client.0 _ll_register_callbacks cb 0x7f65a4013730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47690/0 (socket says 172.21.15.26:47690) 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65aa7fc640 1 -- 172.21.15.26:0/2346533740 learned_addr learned my addr 172.21.15.26:0/2346533740 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65aaffd640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.165+0000 7f65ab7fe640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 -- 172.21.15.26:0/2346533740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 msgr2=0x55ef207c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 -- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 msgr2=0x55ef207b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 -- 172.21.15.26:0/2346533740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef207ad380 con 0x55ef207ace40 2024-04-23T18:23:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65aa7fc640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f65a002a9a0 tx=0x7f65a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f38c8f0753f8579 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a00d6060 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f65a00c5370 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f659c026070 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f65a0048040 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 10 client.16407 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ef207a8960 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f659c026070 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f65a004b060 con 0x55ef207ace40 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 10 client.16407 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.166+0000 7f65b4173740 20 client.16407 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef20831b80 0x55ef20833fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f65b4173740 4 client.16407 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ef20719200 con 0x55ef20831b80 2024-04-23T18:23:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f65aaffd640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef20831b80 0x55ef20833fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f65aaffd640 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef20831b80 0x55ef20833fa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81fa06bf03bc0114 server_cookie=c4986c67d38ba690 in_seq=0 out_seq=0 2024-04-23T18:23:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.167+0000 7f658eff5640 10 client.16407 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f658eff5640 1 -- 172.21.15.26:0/2346533740 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ef20719200 con 0x55ef20831b80 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f658eff5640 10 client.16407 handle_command_reply: tid=0 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 client.16407 shutdown 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 msgr2=0x55ef207c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f65a002a9a0 tx=0x7f65a0000bc0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 shutdown_connections 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55ef20831b80 0x55ef20833fa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef207bbc10 0x55ef207b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f658eff5640 0 client.16407 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55ef207ace40 0x55ef207c3d90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 --2- 172.21.15.26:0/2346533740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef207ac520 0x55ef207c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 >> 172.21.15.26:0/2346533740 conn(0x55ef207a8b30 msgr2=0x55ef207b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.203+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 shutdown_connections 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b4173740 1 -- 172.21.15.26:0/2346533740 wait complete. 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b4173740 20 client.16407 trim_cache size 0 max 16384 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:23:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:23:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 msgr2=0x7f65940a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 0x7f65940a6280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009fd0 tx=0x7f65a4013460 comp rx=0 tx=0).stop 2024-04-23T18:23:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 msgr2=0x7f65ac19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 0x7f65ac19e920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f65a0006010 tx=0x7f65a0004060 comp rx=0 tx=0).stop 2024-04-23T18:23:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 shutdown_connections 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f65940a3dc0 0x7f65940a6280 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65ac103b00 0x7f65ac19e920 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f65ac10c5b0 0x7f65ac198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 --2- 172.21.15.26:0/1938470864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65ac1031e0 0x7f65ac19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.204+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 >> 172.21.15.26:0/1938470864 conn(0x7f65ac077f60 msgr2=0x7f65ac0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.205+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 shutdown_connections 2024-04-23T18:23:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:30.205+0000 7f65b137d640 1 -- 172.21.15.26:0/1938470864 wait complete. 2024-04-23T18:23:30.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 373033 5.69 MB/sec execute 205 sec latency 92.762 ms 2024-04-23T18:23:31.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:30 smithi131 ceph-mon[33041]: pgmap v194: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:30 smithi110 ceph-mon[32585]: pgmap v194: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:30 smithi026 ceph-mon[29364]: pgmap v194: 97 pgs: 97 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 223 op/s 2024-04-23T18:23:31.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 374867 5.70 MB/sec execute 206 sec latency 18.681 ms 2024-04-23T18:23:32.677 INFO:tasks.workunit.client.0.smithi026.stdout: 1 376577 5.73 MB/sec execute 207 sec latency 11.897 ms 2024-04-23T18:23:33.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:32 smithi131 ceph-mon[33041]: pgmap v195: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.8 MiB/s wr, 336 op/s 2024-04-23T18:23:33.260 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:32 smithi026 ceph-mon[29364]: pgmap v195: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.8 MiB/s wr, 336 op/s 2024-04-23T18:23:33.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:32 smithi110 ceph-mon[32585]: pgmap v195: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.8 MiB/s wr, 336 op/s 2024-04-23T18:23:33.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 378347 5.73 MB/sec execute 208 sec latency 19.883 ms 2024-04-23T18:23:34.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 379993 5.76 MB/sec execute 209 sec latency 13.866 ms 2024-04-23T18:23:35.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:34 smithi131 ceph-mon[33041]: pgmap v196: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-04-23T18:23:35.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:34 smithi110 ceph-mon[32585]: pgmap v196: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-04-23T18:23:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:34 smithi026 ceph-mon[29364]: pgmap v196: 97 pgs: 97 active+clean; 205 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-04-23T18:23:35.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 381258 5.76 MB/sec execute 210 sec latency 73.605 ms 2024-04-23T18:23:36.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 383384 5.80 MB/sec execute 211 sec latency 57.157 ms 2024-04-23T18:23:37.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:37 smithi110 ceph-mon[32585]: pgmap v197: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T18:23:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:37 smithi026 ceph-mon[29364]: pgmap v197: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T18:23:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:37 smithi131 ceph-mon[33041]: pgmap v197: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T18:23:37.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 384572 5.80 MB/sec execute 212 sec latency 24.435 ms 2024-04-23T18:23:38.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 386631 5.83 MB/sec execute 213 sec latency 15.279 ms 2024-04-23T18:23:39.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:39 smithi110 ceph-mon[32585]: pgmap v198: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:39 smithi026 ceph-mon[29364]: pgmap v198: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:39.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:39 smithi131 ceph-mon[33041]: pgmap v198: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:39.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 387749 5.82 MB/sec execute 214 sec latency 40.150 ms 2024-04-23T18:23:40.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 390051 5.85 MB/sec execute 215 sec latency 30.352 ms 2024-04-23T18:23:41.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:41 smithi110 ceph-mon[32585]: pgmap v199: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:41.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:41 smithi026 ceph-mon[29364]: pgmap v199: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:41.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:41 smithi131 ceph-mon[33041]: pgmap v199: 97 pgs: 97 active+clean; 212 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:23:41.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 391383 5.85 MB/sec execute 216 sec latency 24.570 ms 2024-04-23T18:23:42.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 393557 5.88 MB/sec execute 217 sec latency 20.241 ms 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.294+0000 7f8a992cb640 1 -- 172.21.15.26:0/1451250012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941010e0 msgr2=0x7f8a941014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.294+0000 7f8a992cb640 1 --2- 172.21.15.26:0/1451250012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941010e0 0x7f8a941014c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009860 tx=0x7f8a8402a340 comp rx=0 tx=0).stop 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 -- 172.21.15.26:0/1451250012 shutdown_connections 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 --2- 172.21.15.26:0/1451250012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a94101a90 0x7f8a94105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 --2- 172.21.15.26:0/1451250012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a9410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 --2- 172.21.15.26:0/1451250012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941010e0 0x7f8a941014c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 -- 172.21.15.26:0/1451250012 >> 172.21.15.26:0/1451250012 conn(0x7f8a940771d0 msgr2=0x7f8a940775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.290 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 -- 172.21.15.26:0/1451250012 shutdown_connections 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 -- 172.21.15.26:0/1451250012 wait complete. 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 Processor -- start 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.295+0000 7f8a992cb640 1 -- start start 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a941010e0 0x7f8a94102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941035f0 0x7f8a941a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a94111110 con 0x7f8a941061b0 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a94110f90 con 0x7f8a941010e0 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a94111290 con 0x7f8a941035f0 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a92ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a941010e0 0x7f8a94102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42182/0 (socket says 172.21.15.26:42182) 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 -- 172.21.15.26:0/3201928209 learned_addr learned my addr 172.21.15.26:0/3201928209 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:43.291 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a937fe640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941035f0 0x7f8a941a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 -- 172.21.15.26:0/3201928209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941035f0 msgr2=0x7f8a941a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941035f0 0x7f8a941a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 -- 172.21.15.26:0/3201928209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a941010e0 msgr2=0x7f8a94102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a941010e0 0x7f8a94102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 -- 172.21.15.26:0/3201928209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a941a6fc0 con 0x7f8a941061b0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a927fc640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800e9f0 tx=0x7f8a8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a7bfff640 1 -- 172.21.15.26:0/3201928209 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8800ccc0 con 0x7f8a941061b0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a941a72b0 con 0x7f8a941061b0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a7bfff640 1 -- 172.21.15.26:0/3201928209 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a880054d0 con 0x7f8a941061b0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.296+0000 7f8a7bfff640 1 -- 172.21.15.26:0/3201928209 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88010430 con 0x7f8a941061b0 2024-04-23T18:23:43.292 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.297+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a941a77b0 con 0x7f8a941061b0 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a7bfff640 1 -- 172.21.15.26:0/3201928209 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a880046c0 con 0x7f8a941061b0 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 Processor -- start 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 -- start start 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4cf40 0x55b09ea572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b09e9b1970 con 0x55b09ea57800 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b09e94e4a0 con 0x55b09ea4c620 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b09e9ad930 con 0x55b09ea4cf40 2024-04-23T18:23:43.299 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a937fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a937fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:57070/0 (socket says 172.21.15.26:57070) 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42194/0 (socket says 172.21.15.26:42194) 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a937fe640 1 -- 172.21.15.26:0/439574793 learned_addr learned my addr 172.21.15.26:0/439574793 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a7bfff640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 0x7f8a7c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.299+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4cf40 0x55b09ea572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 0x7f8a7c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a7bfff640 1 -- 172.21.15.26:0/3201928209 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a880d0ad0 con 0x7f8a941061b0 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 -- 172.21.15.26:0/439574793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4cf40 msgr2=0x55b09ea572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4cf40 0x55b09ea572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 -- 172.21.15.26:0/439574793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 msgr2=0x55b09ea59bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 -- 172.21.15.26:0/439574793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b09e6c4210 con 0x55b09ea4c620 2024-04-23T18:23:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a927fc640 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c005c40 tx=0x7f8a8c00edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ab29fee7fb073e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a937fe640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 0x7f8a7c0a62b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c01c000 tx=0x7f8a8c01e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a797fa640 1 -- 172.21.15.26:0/439574793 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c004050 con 0x55b09ea4c620 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a797fa640 1 -- 172.21.15.26:0/439574793 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a8c021030 con 0x55b09ea4c620 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.300+0000 7f8a797fa640 1 -- 172.21.15.26:0/439574793 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c01dac0 con 0x55b09ea4c620 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/439574793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 msgr2=0x55b09ea4ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c005c40 tx=0x7f8a8c00edb0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/439574793 shutdown_connections 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea57800 0x55b09ea59bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4cf40 0x55b09ea572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/439574793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea4ca00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/439574793 >> 172.21.15.26:0/439574793 conn(0x55b09ea48b30 msgr2=0x55b09ea48f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.301 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/439574793 shutdown_connections 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/439574793 wait complete. 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 -- start start 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.301+0000 7f8a9bfdb740 1 Processor -- start 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4c620 0x55b09ea571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea57800 0x55b09ea54af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b09ea5a3d0 con 0x55b09ea4cf40 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b09e94c0b0 con 0x55b09ea57800 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b09ea5a550 con 0x55b09ea4c620 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 10 client.0 _ll_register_callbacks cb 0x55b09e939d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42208/0 (socket says 172.21.15.26:42208) 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 -- 172.21.15.26:0/79675227 learned_addr learned my addr 172.21.15.26:0/79675227 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a937fe640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea57800 0x55b09ea54af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4c620 0x55b09ea571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 -- 172.21.15.26:0/79675227 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4c620 msgr2=0x55b09ea571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4c620 0x55b09ea571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 -- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea57800 msgr2=0x55b09ea54af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea57800 0x55b09ea54af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 -- 172.21.15.26:0/79675227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b09e790620 con 0x55b09ea4cf40 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a927fc640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c6a0 tx=0x7f8a8800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7098168e5fb8001 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88005680 con 0x55b09ea4cf40 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a880987f0 con 0x55b09ea4cf40 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a8c021030 con 0x55b09ea4cf40 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a8809d1a0 con 0x55b09ea4cf40 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 10 client.16419 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.302+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b09ea66430 con 0x55b09ea4cf40 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a8c021030 con 0x55b09ea4cf40 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8a880a5280 con 0x55b09ea4cf40 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 10 client.16419 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 20 client.16419 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09ea678f0 0x55b09ea69ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 4 client.16419 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b09e9bf390 con 0x55b09ea678f0 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09ea678f0 0x55b09ea69ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09ea678f0 0x55b09ea69ce0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47116d4d75f99066 server_cookie=9428baabab4e1b97 in_seq=0 out_seq=0 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.303+0000 7f8a76ff5640 10 client.16419 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a76ff5640 1 -- 172.21.15.26:0/79675227 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b09e9bf390 con 0x55b09ea678f0 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a76ff5640 10 client.16419 handle_command_reply: tid=0 2024-04-23T18:23:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 client.16419 shutdown 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 msgr2=0x55b09ea544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c6a0 tx=0x7f8a8800ce60 comp rx=0 tx=0).stop 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 shutdown_connections 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09ea678f0 0x55b09ea69ce0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea57800 0x55b09ea54af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea544e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a76ff5640 0 client.16419 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/79675227 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea4c620 0x55b09ea571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 >> 172.21.15.26:0/79675227 conn(0x55b09ea48b30 msgr2=0x55b09ea4e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 shutdown_connections 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.304+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/79675227 wait complete. 2024-04-23T18:23:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.305+0000 7f8a9bfdb740 20 client.16419 trim_cache size 0 max 16384 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 Processor -- start 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 -- start start 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea63290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea6f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941b25c0 con 0x55b09ea4cf40 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941afdd0 con 0x55b09ea4c620 2024-04-23T18:23:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941adc20 con 0x55b09ea57800 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42218/0 (socket says 172.21.15.26:42218) 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 -- 172.21.15.26:0/2968625679 learned_addr learned my addr 172.21.15.26:0/2968625679 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea63290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a937fe640 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea6f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 -- 172.21.15.26:0/2968625679 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 msgr2=0x55b09ea6f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea6f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 -- 172.21.15.26:0/2968625679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 msgr2=0x55b09ea63290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea63290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.320+0000 7f8a927fc640 1 -- 172.21.15.26:0/2968625679 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b09ea6a310 con 0x55b09ea4cf40 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a927fc640 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8a880a55f0 tx=0x7f8a8809afb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937fd2a18bbdea06 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a78ff9640 1 -- 172.21.15.26:0/2968625679 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88065070 con 0x55b09ea4cf40 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a78ff9640 1 -- 172.21.15.26:0/2968625679 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a88005680 con 0x55b09ea4cf40 2024-04-23T18:23:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a78ff9640 1 -- 172.21.15.26:0/2968625679 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a880a0e50 con 0x55b09ea4cf40 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/2968625679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 msgr2=0x55b09ea56d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8a880a55f0 tx=0x7f8a8809afb0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/2968625679 shutdown_connections 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea6f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4cf40 0x55b09ea56d20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/2968625679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4c620 0x55b09ea63290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/2968625679 >> 172.21.15.26:0/2968625679 conn(0x55b09ea48b30 msgr2=0x55b09ea47cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/2968625679 shutdown_connections 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.321+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/2968625679 wait complete. 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 -- start start 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 Processor -- start 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 0x55b09ea5fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea5c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ae290 con 0x55b09ea4c620 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941af700 con 0x55b09ea4cf40 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8a941ae960 con 0x55b09ea57800 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a9bfdb740 10 client.0 _ll_register_callbacks cb 0x7f8a8809c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:42222/0 (socket says 172.21.15.26:42222) 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 -- 172.21.15.26:0/742139728 learned_addr learned my addr 172.21.15.26:0/742139728 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a927fc640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 0x55b09ea5fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a937fe640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea5c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 -- 172.21.15.26:0/742139728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 msgr2=0x55b09ea5c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea5c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 -- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 msgr2=0x55b09ea5fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 0x55b09ea5fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a92ffd640 1 -- 172.21.15.26:0/742139728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b09ea5cde0 con 0x55b09ea4c620 2024-04-23T18:23:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.322+0000 7f8a927fc640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 0x55b09ea5fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a92ffd640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8a84000c00 tx=0x7f8a84005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d652bffe4e7d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a84037040 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8a84004300 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a88005680 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a84057050 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 10 client.16425 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b09eace750 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a88005680 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8a84003b80 con 0x55b09ea4c620 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 10 client.16425 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 20 client.16425 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09eacef30 0x55b09ead1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 4 client.16425 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.323+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b09e9b9200 con 0x55b09eacef30 2024-04-23T18:23:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.324+0000 7f8a927fc640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09eacef30 0x55b09ead1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.324+0000 7f8a927fc640 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09eacef30 0x55b09ead1350 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96f59c6b520c9ff7 server_cookie=36fff8a6623bcf58 in_seq=0 out_seq=0 2024-04-23T18:23:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.324+0000 7f8a76ff5640 10 client.16425 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a76ff5640 1 -- 172.21.15.26:0/742139728 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b09e9b9200 con 0x55b09eacef30 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a76ff5640 10 client.16425 handle_command_reply: tid=0 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 client.16425 shutdown 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 msgr2=0x55b09ea5f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8a84000c00 tx=0x7f8a84005850 comp rx=0 tx=0).stop 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 shutdown_connections 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b09eacef30 0x55b09ead1350 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b09ea4cf40 0x55b09ea5fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b09ea57800 0x55b09ea5c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a76ff5640 0 client.16425 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 --2- 172.21.15.26:0/742139728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b09ea4c620 0x55b09ea5f7a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 >> 172.21.15.26:0/742139728 conn(0x55b09ea48b30 msgr2=0x55b09ea47cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 shutdown_connections 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.356+0000 7f8a9bfdb740 1 -- 172.21.15.26:0/742139728 wait complete. 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a9bfdb740 20 client.16425 trim_cache size 0 max 16384 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:23:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 msgr2=0x7f8a7c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 0x7f8a7c0a62b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c01c000 tx=0x7f8a8c01e000 comp rx=0 tx=0).stop 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 msgr2=0x7f8a941030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800e9f0 tx=0x7f8a8800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 shutdown_connections 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8a7c0a3df0 0x7f8a7c0a62b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a941035f0 0x7f8a941a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8a941061b0 0x7f8a941030b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 --2- 172.21.15.26:0/3201928209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a941010e0 0x7f8a94102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 >> 172.21.15.26:0/3201928209 conn(0x7f8a940771d0 msgr2=0x7f8a9406aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 shutdown_connections 2024-04-23T18:23:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:43.357+0000 7f8a992cb640 1 -- 172.21.15.26:0/3201928209 wait complete. 2024-04-23T18:23:43.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:43 smithi110 ceph-mon[32585]: pgmap v200: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-04-23T18:23:43.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:43 smithi026 ceph-mon[29364]: pgmap v200: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-04-23T18:23:43.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:43 smithi131 ceph-mon[33041]: pgmap v200: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2024-04-23T18:23:43.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 394759 5.88 MB/sec execute 218 sec latency 25.454 ms 2024-04-23T18:23:44.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 396740 5.91 MB/sec execute 219 sec latency 66.228 ms 2024-04-23T18:23:45.293 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:45 smithi131 ceph-mon[33041]: pgmap v201: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-23T18:23:45.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:45 smithi110 ceph-mon[32585]: pgmap v201: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-23T18:23:45.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:45 smithi026 ceph-mon[29364]: pgmap v201: 97 pgs: 97 active+clean; 218 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-23T18:23:45.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 398100 5.91 MB/sec execute 220 sec latency 41.259 ms 2024-04-23T18:23:46.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 400484 5.94 MB/sec execute 221 sec latency 15.680 ms 2024-04-23T18:23:47.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:47 smithi110 ceph-mon[32585]: pgmap v202: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-04-23T18:23:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:47 smithi026 ceph-mon[29364]: pgmap v202: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-04-23T18:23:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:47 smithi131 ceph-mon[33041]: pgmap v202: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 327 op/s 2024-04-23T18:23:47.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 401772 5.95 MB/sec execute 222 sec latency 22.071 ms 2024-04-23T18:23:48.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 403830 5.97 MB/sec execute 223 sec latency 17.663 ms 2024-04-23T18:23:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:49 smithi110 ceph-mon[32585]: pgmap v203: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:49.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:49 smithi026 ceph-mon[29364]: pgmap v203: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:49.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:49 smithi131 ceph-mon[33041]: pgmap v203: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:49.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 405123 5.98 MB/sec execute 224 sec latency 49.548 ms 2024-04-23T18:23:50.503 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-23T18:23:50.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 -- 172.21.15.26:0/1260345012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 msgr2=0x7f0cc010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/1260345012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc010dd40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009ca0 tx=0x7f0cb802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 -- 172.21.15.26:0/1260345012 shutdown_connections 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/1260345012 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc010dd40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/1260345012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc0101a50 0x7f0cc0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/1260345012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc01010a0 0x7f0cc0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 -- 172.21.15.26:0/1260345012 >> 172.21.15.26:0/1260345012 conn(0x7f0cc0077f50 msgr2=0x7f0cc0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 -- 172.21.15.26:0/1260345012 shutdown_connections 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.620+0000 7f0cc83fe640 1 -- 172.21.15.26:0/1260345012 wait complete. 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 Processor -- start 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 -- start start 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 0x7f0cc0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc019cfd0 0x7f0cc019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc01121f0 con 0x7f0cc0106170 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc0112070 con 0x7f0cc019cfd0 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc83fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc0112370 con 0x7f0cc01010a0 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:50.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58104/0 (socket says 172.21.15.26:58104) 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc619a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 0x7f0cc0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 -- 172.21.15.26:0/70531625 learned_addr learned my addr 172.21.15.26:0/70531625 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc699b640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc019cfd0 0x7f0cc019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 -- 172.21.15.26:0/70531625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 msgr2=0x7f0cc0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 0x7f0cc0198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 -- 172.21.15.26:0/70531625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc019cfd0 msgr2=0x7f0cc019d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc019cfd0 0x7f0cc019d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc5999640 1 -- 172.21.15.26:0/70531625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc019dac0 con 0x7f0cc0106170 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.621+0000 7f0cc619a640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 0x7f0cc0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cc5999640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000e9f0 tx=0x7f0cb000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb000ccc0 con 0x7f0cc0106170 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc019ddb0 con 0x7f0cc0106170 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc01a55a0 con 0x7f0cc0106170 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cb00054d0 con 0x7f0cc0106170 2024-04-23T18:23:50.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.622+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0010430 con 0x7f0cc0106170 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.623+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc0068d70 con 0x7f0cc0106170 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.624+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0cb00046c0 con 0x7f0cc0106170 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.625+0000 7f0cb77fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 0x7f0cac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.625+0000 7f0cc619a640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 0x7f0cac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.625+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0cb00d0ad0 con 0x7f0cc0106170 2024-04-23T18:23:50.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.625+0000 7f0cc619a640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 0x7f0cac0a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009fc0 tx=0x7f0cbc005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.630+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0cb009d140 con 0x7f0cc0106170 2024-04-23T18:23:50.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 407277 6.00 MB/sec execute 225 sec latency 52.492 ms 2024-04-23T18:23:50.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.874+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cc0109400 con 0x7f0cc0106170 2024-04-23T18:23:50.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.874+0000 7f0cb77fe640 1 -- 172.21.15.26:0/70531625 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11893 (secure 0 0 0) 0x7f0cb009ca40 con 0x7f0cc0106170 2024-04-23T18:23:50.868 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:23:50.868 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:22:59.417042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:23:50.868 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.876+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 msgr2=0x7f0cac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.876+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 0x7f0cac0a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009fc0 tx=0x7f0cbc005f90 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.876+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 msgr2=0x7f0cc0198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.876+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000e9f0 tx=0x7f0cb000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 shutdown_connections 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0cac0a3d20 0x7f0cac0a61e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cc019cfd0 0x7f0cc019d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0cc0106170 0x7f0cc0198c50 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 --2- 172.21.15.26:0/70531625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0cc01010a0 0x7f0cc0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 >> 172.21.15.26:0/70531625 conn(0x7f0cc0077f50 msgr2=0x7f0cc00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:50.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 shutdown_connections 2024-04-23T18:23:50.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:50.877+0000 7f0cc83fe640 1 -- 172.21.15.26:0/70531625 wait complete. 2024-04-23T18:23:50.881 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-23T18:23:50.881 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-23T18:23:50.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 -- 172.21.15.26:0/650474065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 msgr2=0x7fd08010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:50.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 --2- 172.21.15.26:0/650474065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd08010dd40 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd078009ca0 tx=0x7fd07802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:23:50.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 -- 172.21.15.26:0/650474065 shutdown_connections 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 --2- 172.21.15.26:0/650474065 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd08010dd40 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 --2- 172.21.15.26:0/650474065 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd080101a50 0x7fd080105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 --2- 172.21.15.26:0/650474065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 0x7fd080101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 -- 172.21.15.26:0/650474065 >> 172.21.15.26:0/650474065 conn(0x7fd080077f50 msgr2=0x7fd080078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 -- 172.21.15.26:0/650474065 shutdown_connections 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.004+0000 7fd087920640 1 -- 172.21.15.26:0/650474065 wait complete. 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 Processor -- start 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 -- start start 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 0x7fd0801986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd08019cf80 0x7fd08019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801110e0 con 0x7fd080106170 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd080110f60 con 0x7fd0801010a0 2024-04-23T18:23:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd087920640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd080111260 con 0x7fd08019cf80 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd084ebb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd084ebb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58130/0 (socket says 172.21.15.26:58130) 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd084ebb640 1 -- 172.21.15.26:0/1901303596 learned_addr learned my addr 172.21.15.26:0/1901303596 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd085ebd640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd08019cf80 0x7fd08019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.005+0000 7fd0856bc640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 0x7fd0801986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 -- 172.21.15.26:0/1901303596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd08019cf80 msgr2=0x7fd08019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd08019cf80 0x7fd08019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 -- 172.21.15.26:0/1901303596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 msgr2=0x7fd0801986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 0x7fd0801986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 -- 172.21.15.26:0/1901303596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08019da70 con 0x7fd080106170 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd084ebb640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9f0 tx=0x7fd07000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd0767fc640 1 -- 172.21.15.26:0/1901303596 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07000ccc0 con 0x7fd080106170 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd0767fc640 1 -- 172.21.15.26:0/1901303596 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0700054d0 con 0x7fd080106170 2024-04-23T18:23:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08019dd60 con 0x7fd080106170 2024-04-23T18:23:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd0767fc640 1 -- 172.21.15.26:0/1901303596 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070010430 con 0x7fd080106170 2024-04-23T18:23:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.006+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0801a5550 con 0x7fd080106170 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd0767fc640 1 -- 172.21.15.26:0/1901303596 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0700046c0 con 0x7fd080106170 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 Processor -- start 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 -- start start 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2837f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5598e278e370 con 0x5598e28384a0 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5598e272b2e0 con 0x5598e2828f80 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd08a634740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5598e278a330 con 0x5598e28298a0 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd084ebb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:56372/0 (socket says 172.21.15.26:56372) 2024-04-23T18:23:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd085ebd640 1 -- 172.21.15.26:0/2295240636 learned_addr learned my addr 172.21.15.26:0/2295240636 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd0767fc640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 0x7fd06c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2837f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.009+0000 7fd085ebd640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 0x7fd06c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 -- 172.21.15.26:0/2295240636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 msgr2=0x5598e2837f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2837f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 -- 172.21.15.26:0/2295240636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 msgr2=0x5598e283a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd0767fc640 1 -- 172.21.15.26:0/1901303596 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd0700d0a40 con 0x7fd080106170 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 -- 172.21.15.26:0/2295240636 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e256cd80 con 0x5598e2828f80 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd084ebb640 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd078035000 tx=0x7fd078031d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ee66838362a55f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd085ebd640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 0x7fd06c0a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd078031e00 tx=0x7fd07804a000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd06b7fe640 1 -- 172.21.15.26:0/2295240636 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078003b70 con 0x5598e2828f80 2024-04-23T18:23:51.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd06b7fe640 1 -- 172.21.15.26:0/2295240636 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078036070 con 0x5598e2828f80 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.010+0000 7fd06b7fe640 1 -- 172.21.15.26:0/2295240636 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07802f660 con 0x5598e2828f80 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- 172.21.15.26:0/2295240636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 msgr2=0x5598e2829360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd078035000 tx=0x7fd078031d90 comp rx=0 tx=0).stop 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- 172.21.15.26:0/2295240636 shutdown_connections 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2837f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 --2- 172.21.15.26:0/2295240636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2829360 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- 172.21.15.26:0/2295240636 >> 172.21.15.26:0/2295240636 conn(0x5598e28253c0 msgr2=0x5598e28257a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- 172.21.15.26:0/2295240636 shutdown_connections 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- 172.21.15.26:0/2295240636 wait complete. 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 -- start start 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.011+0000 7fd08a634740 1 Processor -- start 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2835970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5598e283b100 con 0x5598e28384a0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5598e2728fb0 con 0x5598e2828f80 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5598e283b280 con 0x5598e28298a0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd08a634740 10 client.0 _ll_register_callbacks cb 0x5598e27289a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd0856bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58156/0 (socket says 172.21.15.26:58156) 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 -- 172.21.15.26:0/2175609281 learned_addr learned my addr 172.21.15.26:0/2175609281 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd084ebb640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2835970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 -- 172.21.15.26:0/2175609281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 msgr2=0x5598e2835970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2835970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 -- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 msgr2=0x5598e2835430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 -- 172.21.15.26:0/2175609281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e25cd510 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd084ebb640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2835970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd07802ade0 tx=0x7fd078004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9206f713c675091d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.012+0000 7fd068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078061060 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd078003970 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd078036070 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd078003bb0 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 10 client.16443 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5598e2460670 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078036070 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd07802fb10 con 0x5598e28384a0 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 10 client.16443 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 10 client.16443 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 20 client.16443 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e2848560 0x5598e284a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 4 client.16443 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.013+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598e279bd90 con 0x5598e2848560 2024-04-23T18:23:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.014+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e2848560 0x5598e284a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.014+0000 7fd068ff9640 10 client.16443 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.014+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e2848560 0x5598e284a950 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8aafefa2202256 server_cookie=9355b6af7344ffe7 in_seq=0 out_seq=0 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd068ff9640 1 -- 172.21.15.26:0/2175609281 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598e279bd90 con 0x5598e2848560 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd068ff9640 10 client.16443 handle_command_reply: tid=0 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 client.16443 shutdown 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 msgr2=0x5598e2835eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd07802ade0 tx=0x7fd078004440 comp rx=0 tx=0).stop 2024-04-23T18:23:51.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 shutdown_connections 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e2848560 0x5598e284a950 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e2835eb0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2835970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd068ff9640 0 client.16443 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 --2- 172.21.15.26:0/2175609281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 >> 172.21.15.26:0/2175609281 conn(0x5598e28253c0 msgr2=0x5598e282b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 shutdown_connections 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 1 -- 172.21.15.26:0/2175609281 wait complete. 2024-04-23T18:23:51.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.015+0000 7fd08a634740 20 client.16443 trim_cache size 0 max 16384 2024-04-23T18:23:51.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 Processor -- start 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 -- start start 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2850090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801ae1e0 con 0x5598e28384a0 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801ab9f0 con 0x5598e2828f80 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.037+0000 7fd08a634740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801a9880 con 0x5598e28298a0 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd084ebb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2850090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd0856bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58170/0 (socket says 172.21.15.26:58170) 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 -- 172.21.15.26:0/3148653781 learned_addr learned my addr 172.21.15.26:0/3148653781 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 -- 172.21.15.26:0/3148653781 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 msgr2=0x5598e2850090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2850090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 -- 172.21.15.26:0/3148653781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 msgr2=0x5598e2835cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 -- 172.21.15.26:0/3148653781 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598e2834400 con 0x5598e28384a0 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd085ebd640 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd078002230 tx=0x7fd07802f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4800479c74bb1931 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd06affd640 1 -- 172.21.15.26:0/3148653781 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078061040 con 0x5598e28384a0 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd06affd640 1 -- 172.21.15.26:0/3148653781 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078036040 con 0x5598e28384a0 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.038+0000 7fd06affd640 1 -- 172.21.15.26:0/3148653781 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078003d50 con 0x5598e28384a0 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- 172.21.15.26:0/3148653781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 msgr2=0x5598e28505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd078002230 tx=0x7fd07802f500 comp rx=0 tx=0).stop 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- 172.21.15.26:0/3148653781 shutdown_connections 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e28505d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e28298a0 0x5598e2850090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 --2- 172.21.15.26:0/3148653781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e2828f80 0x5598e2835cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- 172.21.15.26:0/3148653781 >> 172.21.15.26:0/3148653781 conn(0x5598e28253c0 msgr2=0x5598e282a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- 172.21.15.26:0/3148653781 shutdown_connections 2024-04-23T18:23:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- 172.21.15.26:0/3148653781 wait complete. 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 -- start start 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.039+0000 7fd08a634740 1 Processor -- start 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e28298a0 0x5598e283d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801a9ed0 con 0x5598e28384a0 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801ab320 con 0x5598e28298a0 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801aa580 con 0x5598e2828f80 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd08a634740 10 client.0 _ll_register_callbacks cb 0x7fd078009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd0856bc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58184/0 (socket says 172.21.15.26:58184) 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd0856bc640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:34392/0 (socket says 172.21.15.26:34392) 2024-04-23T18:23:51.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd084ebb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e28298a0 0x5598e283d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 -- 172.21.15.26:0/2350721680 learned_addr learned my addr 172.21.15.26:0/2350721680 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 -- 172.21.15.26:0/2350721680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 msgr2=0x5598e2840700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 -- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e28298a0 msgr2=0x5598e283d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e28298a0 0x5598e283d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 -- 172.21.15.26:0/2350721680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598e283df60 con 0x5598e28384a0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd085ebd640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd07802fee0 tx=0x7fd078004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337527ff53560670 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.040+0000 7fd068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078004b40 con 0x5598e28384a0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd078004ce0 con 0x5598e28384a0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd078036040 con 0x5598e28384a0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd078030a90 con 0x5598e28384a0 2024-04-23T18:23:51.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 10 client.16449 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5598e28af610 con 0x5598e28384a0 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078036040 con 0x5598e28384a0 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd078030c30 con 0x5598e28384a0 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 10 client.16449 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 10 client.16449 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 20 client.16449 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e28afb10 0x5598e28b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 4 client.16449 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.041+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x5598e2795c00 con 0x5598e28afb10 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.042+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e28afb10 0x5598e28b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.042+0000 7fd068ff9640 10 client.16449 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:51.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.042+0000 7fd0856bc640 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e28afb10 0x5598e28b1f30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7785cb2205ec62 server_cookie=a3b0d63d4ec1d9ba in_seq=0 out_seq=0 2024-04-23T18:23:51.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd068ff9640 1 -- 172.21.15.26:0/2350721680 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5598e2795c00 con 0x5598e28afb10 2024-04-23T18:23:51.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd068ff9640 10 client.16449 handle_command_reply: tid=0 2024-04-23T18:23:51.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 client.16449 shutdown 2024-04-23T18:23:51.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 msgr2=0x5598e283da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd07802fee0 tx=0x7fd078004440 comp rx=0 tx=0).stop 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 shutdown_connections 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5598e28afb10 0x5598e28b1f30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5598e28298a0 0x5598e283d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5598e28384a0 0x5598e283da20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 --2- 172.21.15.26:0/2350721680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598e2828f80 0x5598e2840700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 >> 172.21.15.26:0/2350721680 conn(0x5598e28253c0 msgr2=0x5598e282b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.125+0000 7fd068ff9640 0 client.16449 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.126+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 shutdown_connections 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.126+0000 7fd08a634740 1 -- 172.21.15.26:0/2350721680 wait complete. 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.126+0000 7fd08a634740 20 client.16449 trim_cache size 0 max 16384 2024-04-23T18:23:51.120 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 37, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 30, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "27958699": { 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 30, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:23:51.121 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:23:51.122 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 msgr2=0x7fd06c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 0x7fd06c0a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd078031e00 tx=0x7fd07804a000 comp rx=0 tx=0).stop 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 msgr2=0x7fd080198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9f0 tx=0x7fd07000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 shutdown_connections 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd06c0a3da0 0x7fd06c0a6260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd08019cf80 0x7fd08019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd080106170 0x7fd080198c00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 --2- 172.21.15.26:0/1901303596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0801010a0 0x7fd0801986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 >> 172.21.15.26:0/1901303596 conn(0x7fd080077f50 msgr2=0x7fd080069440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 shutdown_connections 2024-04-23T18:23:51.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:23:51.127+0000 7fd087920640 1 -- 172.21.15.26:0/1901303596 wait complete. 2024-04-23T18:23:51.131 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:51 smithi026 ceph-mon[29364]: pgmap v204: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:51.131 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:51 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/70531625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:23:51.132 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '27958699', quiesce took 0.6 seconds. Will release after: 10 2024-04-23T18:23:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:51 smithi110 ceph-mon[32585]: pgmap v204: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:51 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/70531625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:23:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:51 smithi131 ceph-mon[33041]: pgmap v204: 97 pgs: 97 active+clean; 225 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-04-23T18:23:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:51 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/70531625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:23:51.678 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 6.01 MB/sec execute 226 sec latency 578.754 ms 2024-04-23T18:23:52.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.98 MB/sec execute 227 sec latency 1578.903 ms 2024-04-23T18:23:53.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:53 smithi110 ceph-mon[32585]: pgmap v205: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 354 op/s 2024-04-23T18:23:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:53 smithi026 ceph-mon[29364]: pgmap v205: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 354 op/s 2024-04-23T18:23:53.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:53 smithi131 ceph-mon[33041]: pgmap v205: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 354 op/s 2024-04-23T18:23:53.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.95 MB/sec execute 228 sec latency 2579.056 ms 2024-04-23T18:23:54.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.93 MB/sec execute 229 sec latency 3579.137 ms 2024-04-23T18:23:55.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:55 smithi110 ceph-mon[32585]: pgmap v206: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-23T18:23:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:55 smithi026 ceph-mon[29364]: pgmap v206: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-23T18:23:55.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:55 smithi131 ceph-mon[33041]: pgmap v206: 97 pgs: 97 active+clean; 230 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-23T18:23:55.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.90 MB/sec execute 230 sec latency 4579.282 ms 2024-04-23T18:23:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 -- 172.21.15.26:0/974452484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341021d0 msgr2=0x7f32341025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 --2- 172.21.15.26:0/974452484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341021d0 0x7f32341025b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f321c009900 tx=0x7f321c02a410 comp rx=0 tx=0).stop 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 -- 172.21.15.26:0/974452484 shutdown_connections 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 --2- 172.21.15.26:0/974452484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3234102b80 0x7f3234106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 --2- 172.21.15.26:0/974452484 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341072a0 0x7f323410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 --2- 172.21.15.26:0/974452484 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341021d0 0x7f32341025b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 -- 172.21.15.26:0/974452484 >> 172.21.15.26:0/974452484 conn(0x7f32340793e0 msgr2=0x7f32340797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 -- 172.21.15.26:0/974452484 shutdown_connections 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.456+0000 7f323a89a640 1 -- 172.21.15.26:0/974452484 wait complete. 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.457+0000 7f323a89a640 1 Processor -- start 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.457+0000 7f323a89a640 1 -- start start 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.457+0000 7f323a89a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341021d0 0x7f3234198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.457+0000 7f323a89a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f323419cf20 0x7f323419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3234112260 con 0x7f32341072a0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32341120e0 con 0x7f323419cf20 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f32341123e0 con 0x7f32341021d0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58198/0 (socket says 172.21.15.26:58198) 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 -- 172.21.15.26:0/721817712 learned_addr learned my addr 172.21.15.26:0/721817712 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f3233fff640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341021d0 0x7f3234198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f3238e37640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f323419cf20 0x7f323419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 -- 172.21.15.26:0/721817712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341021d0 msgr2=0x7f3234198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341021d0 0x7f3234198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 -- 172.21.15.26:0/721817712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f323419cf20 msgr2=0x7f323419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f323419cf20 0x7f323419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 -- 172.21.15.26:0/721817712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323419da10 con 0x7f32341072a0 2024-04-23T18:23:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32337fe640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f322400c970 tx=0x7f322400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32317fa640 1 -- 172.21.15.26:0/721817712 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224014070 con 0x7f32341072a0 2024-04-23T18:23:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32317fa640 1 -- 172.21.15.26:0/721817712 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3224004440 con 0x7f32341072a0 2024-04-23T18:23:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323419dd00 con 0x7f32341072a0 2024-04-23T18:23:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f32317fa640 1 -- 172.21.15.26:0/721817712 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224003e50 con 0x7f32341072a0 2024-04-23T18:23:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.458+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32341a54f0 con 0x7f32341072a0 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f32317fa640 1 -- 172.21.15.26:0/721817712 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3224017070 con 0x7f32341072a0 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 Processor -- start 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 -- start start 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e175a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e182bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f97e0da970 con 0x55f97e175f40 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f97e0774a0 con 0x55f97e180800 2024-04-23T18:23:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f323d5b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f97e0d6930 con 0x55f97e175620 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58208/0 (socket says 172.21.15.26:58208) 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.461+0000 7f3233fff640 1 -- 172.21.15.26:0/3811766855 learned_addr learned my addr 172.21.15.26:0/3811766855 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f32317fa640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 0x7f32200a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f32337fe640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e182bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3238e37640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 0x7f32200a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3238e37640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e175a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 -- 172.21.15.26:0/3811766855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 msgr2=0x55f97e175a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e175a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 -- 172.21.15.26:0/3811766855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 msgr2=0x55f97e182bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.462 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e182bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f32317fa640 1 -- 172.21.15.26:0/721817712 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f32240d0410 con 0x7f32341072a0 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 -- 172.21.15.26:0/3811766855 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f97dded210 con 0x55f97e175f40 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3238e37640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e175a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3233fff640 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f321c02a930 tx=0x7f321c031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64731cf6f11a73ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f3238e37640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 0x7f32200a6420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f322c00a4c0 tx=0x7f322c0095f0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f322a7fc640 1 -- 172.21.15.26:0/3811766855 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c03f070 con 0x55f97e175f40 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f322a7fc640 1 -- 172.21.15.26:0/3811766855 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321c0316a0 con 0x55f97e175f40 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.462+0000 7f322a7fc640 1 -- 172.21.15.26:0/3811766855 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c035070 con 0x55f97e175f40 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- 172.21.15.26:0/3811766855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 msgr2=0x55f97e1802c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f321c02a930 tx=0x7f321c031cd0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- 172.21.15.26:0/3811766855 shutdown_connections 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e182bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1802c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3811766855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e175a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- 172.21.15.26:0/3811766855 >> 172.21.15.26:0/3811766855 conn(0x55f97e171b30 msgr2=0x55f97e171f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- 172.21.15.26:0/3811766855 shutdown_connections 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- 172.21.15.26:0/3811766855 wait complete. 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.463+0000 7f323d5b4740 1 -- start start 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 Processor -- start 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e17d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e17d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f97e1833d0 con 0x55f97e175620 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f97e0750b0 con 0x55f97e175f40 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f97e183550 con 0x55f97e180800 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f323d5b4740 10 client.0 _ll_register_callbacks cb 0x55f97e062d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58222/0 (socket says 172.21.15.26:58222) 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 -- 172.21.15.26:0/629137461 learned_addr learned my addr 172.21.15.26:0/629137461 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f32337fe640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e17d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3238e37640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e17d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 -- 172.21.15.26:0/629137461 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 msgr2=0x55f97e17d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e17d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 -- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 msgr2=0x55f97e17d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e17d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.464+0000 7f3233fff640 1 -- 172.21.15.26:0/629137461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f97deb9620 con 0x55f97e175620 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f3233fff640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f321c009900 tx=0x7f321c032fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e72a95659cd8d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c04a070 con 0x55f97e175620 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f321c004bb0 con 0x55f97e175620 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f321c0316a0 con 0x55f97e175620 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f321c056050 con 0x55f97e175620 2024-04-23T18:23:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 10 client.16461 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f97e18f270 con 0x55f97e175620 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321c0316a0 con 0x55f97e175620 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f321c043400 con 0x55f97e175620 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 10 client.16461 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.465+0000 7f323d5b4740 20 client.16461 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1905c0 0x55f97e1929b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f323d5b4740 4 client.16461 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f97e0e8390 con 0x55f97e1905c0 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f32337fe640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1905c0 0x55f97e1929b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f32137fe640 10 client.16461 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.466+0000 7f32337fe640 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1905c0 0x55f97e1929b0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bedbf6e3baeedadf server_cookie=d4ffcda1782e8028 in_seq=0 out_seq=0 2024-04-23T18:23:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f32137fe640 1 -- 172.21.15.26:0/629137461 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f97e0e8390 con 0x55f97e1905c0 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f32137fe640 10 client.16461 handle_command_reply: tid=0 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 client.16461 shutdown 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 msgr2=0x55f97e180150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f321c009900 tx=0x7f321c032fa0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 shutdown_connections 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1905c0 0x55f97e1929b0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e17d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f32137fe640 0 client.16461 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e17d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 --2- 172.21.15.26:0/629137461 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e180150 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 >> 172.21.15.26:0/629137461 conn(0x55f97e171b30 msgr2=0x55f97e177860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 shutdown_connections 2024-04-23T18:23:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 1 -- 172.21.15.26:0/629137461 wait complete. 2024-04-23T18:23:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.467+0000 7f323d5b4740 20 client.16461 trim_cache size 0 max 16384 2024-04-23T18:23:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.482+0000 7f323d5b4740 1 Processor -- start 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.482+0000 7f323d5b4740 1 -- start start 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e170b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f32341b0340 con 0x55f97e175620 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32341adb50 con 0x55f97e175f40 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f323d5b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ab9c0 con 0x55f97e180800 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58238/0 (socket says 172.21.15.26:58238) 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 -- 172.21.15.26:0/3510347423 learned_addr learned my addr 172.21.15.26:0/3510347423 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f32337fe640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e170b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3238e37640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 -- 172.21.15.26:0/3510347423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 msgr2=0x55f97e197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 -- 172.21.15.26:0/3510347423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 msgr2=0x55f97e170b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e170b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 -- 172.21.15.26:0/3510347423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f97e1834f0 con 0x55f97e175620 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3238e37640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3233fff640 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f321c02ab40 tx=0x7f321c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca240b52e03a1ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3229ffb640 1 -- 172.21.15.26:0/3510347423 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c04a070 con 0x55f97e175620 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3229ffb640 1 -- 172.21.15.26:0/3510347423 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321c0439a0 con 0x55f97e175620 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.483+0000 7f3229ffb640 1 -- 172.21.15.26:0/3510347423 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c03f030 con 0x55f97e175620 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- 172.21.15.26:0/3510347423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 msgr2=0x55f97e18c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f321c02ab40 tx=0x7f321c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- 172.21.15.26:0/3510347423 shutdown_connections 2024-04-23T18:23:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e175f40 0x55f97e170b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e180800 0x55f97e197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 --2- 172.21.15.26:0/3510347423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175620 0x55f97e18c290 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- 172.21.15.26:0/3510347423 >> 172.21.15.26:0/3510347423 conn(0x55f97e171b30 msgr2=0x55f97e170420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- 172.21.15.26:0/3510347423 shutdown_connections 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- 172.21.15.26:0/3510347423 wait complete. 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.484+0000 7f323d5b4740 1 -- start start 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 Processor -- start 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e185c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e1ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ac010 con 0x55f97e175f40 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ad480 con 0x55f97e180800 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ac6e0 con 0x55f97e175620 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f323d5b4740 10 client.0 _ll_register_callbacks cb 0x7f321c032ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:58250/0 (socket says 172.21.15.26:58250) 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 -- 172.21.15.26:0/2182630170 learned_addr learned my addr 172.21.15.26:0/2182630170 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f3238e37640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e1ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f3233fff640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e185c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 -- 172.21.15.26:0/2182630170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 msgr2=0x55f97e185c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e185c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 -- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 msgr2=0x55f97e1ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e1ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.485+0000 7f32337fe640 1 -- 172.21.15.26:0/2182630170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f97e1ede20 con 0x55f97e175f40 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32337fe640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f322409c370 tx=0x7f32240d2920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c49ca507017484 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322409c3e0 con 0x55f97e175f40 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f32240cde10 con 0x55f97e175f40 2024-04-23T18:23:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f321c0439a0 con 0x55f97e175f40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f32240a0790 con 0x55f97e175f40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 10 client.16467 fetch_fsmap learned FSMap version 37 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f97e1f63b0 con 0x55f97e175f40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f321c0439a0 con 0x55f97e175f40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3224098130 con 0x55f97e175f40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 10 client.16467 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 10 client.16467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 20 client.16467 populate_metadata read hostname 'smithi026' 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1f6b40 0x55f97e1f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 4 client.16467 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.486+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f97e0e2200 con 0x55f97e1f6b40 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.487+0000 7f3233fff640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1f6b40 0x55f97e1f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.487+0000 7f3233fff640 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1f6b40 0x55f97e1f8f60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1d69a45791d1c server_cookie=fdf82fb1ea29b619 in_seq=0 out_seq=0 2024-04-23T18:23:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.487+0000 7f32137fe640 10 client.16467 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f32137fe640 1 -- 172.21.15.26:0/2182630170 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f97e0e2200 con 0x55f97e1f6b40 2024-04-23T18:23:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f32137fe640 10 client.16467 handle_command_reply: tid=0 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 client.16467 shutdown 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 msgr2=0x55f97e1861b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f322409c370 tx=0x7f32240d2920 comp rx=0 tx=0).stop 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 shutdown_connections 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f97e1f6b40 0x55f97e1f8f60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f97e180800 0x55f97e1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f97e175f40 0x55f97e1861b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f32137fe640 0 client.16467 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 --2- 172.21.15.26:0/2182630170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f97e175620 0x55f97e185c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 >> 172.21.15.26:0/2182630170 conn(0x55f97e171b30 msgr2=0x55f97e170cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 shutdown_connections 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 1 -- 172.21.15.26:0/2182630170 wait complete. 2024-04-23T18:23:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.519+0000 7f323d5b4740 20 client.16467 trim_cache size 0 max 16384 2024-04-23T18:23:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:23:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:23:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:23:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 msgr2=0x7f32200a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 0x7f32200a6420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f322c00a4c0 tx=0x7f322c0095f0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 msgr2=0x7f3234198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f322400c970 tx=0x7f322400ce30 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 shutdown_connections 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f32200a3f60 0x7f32200a6420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f323419cf20 0x7f323419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f32341072a0 0x7f3234198ba0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 --2- 172.21.15.26:0/721817712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32341021d0 0x7f3234198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 >> 172.21.15.26:0/721817712 conn(0x7f32340793e0 msgr2=0x7f32340ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 shutdown_connections 2024-04-23T18:23:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:23:56.520+0000 7f323a89a640 1 -- 172.21.15.26:0/721817712 wait complete. 2024-04-23T18:23:56.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.88 MB/sec execute 231 sec latency 5579.371 ms 2024-04-23T18:23:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:57 smithi110 ceph-mon[32585]: pgmap v207: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 286 op/s 2024-04-23T18:23:57.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:57 smithi026 ceph-mon[29364]: pgmap v207: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 286 op/s 2024-04-23T18:23:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:57 smithi131 ceph-mon[33041]: pgmap v207: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 286 op/s 2024-04-23T18:23:57.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.85 MB/sec execute 232 sec latency 6579.488 ms 2024-04-23T18:23:58.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.83 MB/sec execute 233 sec latency 7579.568 ms 2024-04-23T18:23:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:23:59 smithi110 ceph-mon[32585]: pgmap v208: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:23:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:23:59 smithi026 ceph-mon[29364]: pgmap v208: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:23:59.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:23:59 smithi131 ceph-mon[33041]: pgmap v208: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:23:59.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.80 MB/sec execute 234 sec latency 8579.676 ms 2024-04-23T18:24:00.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408170 5.78 MB/sec execute 235 sec latency 9579.791 ms 2024-04-23T18:24:01.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- 172.21.15.26:0/235279520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 msgr2=0x7f676c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 --2- 172.21.15.26:0/235279520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c102580 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6764009900 tx=0x7f676402a410 comp rx=0 tx=0).stop 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- 172.21.15.26:0/235279520 shutdown_connections 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 --2- 172.21.15.26:0/235279520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 0x7f676c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 --2- 172.21.15.26:0/235279520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c102b50 0x7f676c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 --2- 172.21.15.26:0/235279520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c102580 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- 172.21.15.26:0/235279520 >> 172.21.15.26:0/235279520 conn(0x7f676c0fd9e0 msgr2=0x7f676c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- 172.21.15.26:0/235279520 shutdown_connections 2024-04-23T18:24:01.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- 172.21.15.26:0/235279520 wait complete. 2024-04-23T18:24:01.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 Processor -- start 2024-04-23T18:24:01.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.246+0000 7f6773a74640 1 -- start start 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 0x7f676c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c074cd0 0x7f676c1a6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f676c112250 con 0x7f676c1021a0 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f676c1120d0 con 0x7f676c107270 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f676c1123d0 con 0x7f676c074cd0 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54924/0 (socket says 172.21.15.26:54924) 2024-04-23T18:24:01.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 -- 172.21.15.26:0/4159715038 learned_addr learned my addr 172.21.15.26:0/4159715038 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f677100f640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 0x7f676c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6772011640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c074cd0 0x7f676c1a6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 -- 172.21.15.26:0/4159715038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c074cd0 msgr2=0x7f676c1a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c074cd0 0x7f676c1a6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 -- 172.21.15.26:0/4159715038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 msgr2=0x7f676c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 0x7f676c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 -- 172.21.15.26:0/4159715038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c1a70c0 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6771810640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6764006010 tx=0x7f67640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676403f030 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6764003800 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676c1a7350 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764035070 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.247+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f676c1a7890 con 0x7f676c1021a0 2024-04-23T18:24:01.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.248+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676c068d70 con 0x7f676c1021a0 2024-04-23T18:24:01.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.249+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6764003cb0 con 0x7f676c1021a0 2024-04-23T18:24:01.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.250+0000 7f6762ffd640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 0x7f67580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.250+0000 7f677100f640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 0x7f67580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.250+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f67640f1190 con 0x7f676c1021a0 2024-04-23T18:24:01.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.251+0000 7f677100f640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 0x7f67580a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f675c005fd0 tx=0x7f675c005e20 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.255+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67640bd860 con 0x7f676c1021a0 2024-04-23T18:24:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:01 smithi110 ceph-mon[32585]: pgmap v209: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:24:01.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:01 smithi026 ceph-mon[29364]: pgmap v209: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:24:01.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.480+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f676c10a4a0 con 0x7f676c1021a0 2024-04-23T18:24:01.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.481+0000 7f6762ffd640 1 -- 172.21.15.26:0/4159715038 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11893 (secure 0 0 0) 0x7f67640bd160 con 0x7f676c1021a0 2024-04-23T18:24:01.474 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:24:01.474 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:22:59.417042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:24:01.474 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2024-04-23T18:24:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 msgr2=0x7f67580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 0x7f67580a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f675c005fd0 tx=0x7f675c005e20 comp rx=0 tx=0).stop 2024-04-23T18:24:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 msgr2=0x7f676c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6764006010 tx=0x7f67640042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 shutdown_connections 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f67580a3da0 0x7f67580a6260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f676c107270 0x7f676c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f676c074cd0 0x7f676c1a6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 --2- 172.21.15.26:0/4159715038 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f676c1021a0 0x7f676c074250 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 >> 172.21.15.26:0/4159715038 conn(0x7f676c0fd9e0 msgr2=0x7f676c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 shutdown_connections 2024-04-23T18:24:01.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.483+0000 7f6773a74640 1 -- 172.21.15.26:0/4159715038 wait complete. 2024-04-23T18:24:01.486 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 27958699 --release --await' 2024-04-23T18:24:01.487 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 27958699 --release --await 2024-04-23T18:24:01.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:01 smithi131 ceph-mon[33041]: pgmap v209: 97 pgs: 97 active+clean; 239 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-23T18:24:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.602+0000 7f39506cf640 1 -- 172.21.15.26:0/3029021365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 msgr2=0x7f394810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- 172.21.15.26:0/3029021365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394810e9b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3940009ca0 tx=0x7f394002a9e0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- 172.21.15.26:0/3029021365 shutdown_connections 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- 172.21.15.26:0/3029021365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3948103af0 0x7f394810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- 172.21.15.26:0/3029021365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394810e9b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- 172.21.15.26:0/3029021365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39481031d0 0x7f39481035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- 172.21.15.26:0/3029021365 >> 172.21.15.26:0/3029021365 conn(0x7f3948077f50 msgr2=0x7f3948078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- 172.21.15.26:0/3029021365 shutdown_connections 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- 172.21.15.26:0/3029021365 wait complete. 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 Processor -- start 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- start start 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39481031d0 0x7f3948198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3948103af0 0x7f3948198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f39481121a0 con 0x7f394810c510 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3948112020 con 0x7f39481031d0 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.603+0000 7f39506cf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3948112320 con 0x7f3948103af0 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54948/0 (socket says 172.21.15.26:54948) 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 -- 172.21.15.26:0/2070425963 learned_addr learned my addr 172.21.15.26:0/2070425963 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394e46b640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39481031d0 0x7f3948198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3948103af0 0x7f3948198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 -- 172.21.15.26:0/2070425963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3948103af0 msgr2=0x7f3948198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3948103af0 0x7f3948198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 -- 172.21.15.26:0/2070425963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39481031d0 msgr2=0x7f3948198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39481031d0 0x7f3948198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 -- 172.21.15.26:0/2070425963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394819d650 con 0x7f394810c510 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f394ec6c640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3940006010 tx=0x7f394002fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f393f7fe640 1 -- 172.21.15.26:0/2070425963 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940033040 con 0x7f394810c510 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f393f7fe640 1 -- 172.21.15.26:0/2070425963 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394002e7a0 con 0x7f394810c510 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f394819d8e0 con 0x7f394810c510 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f393f7fe640 1 -- 172.21.15.26:0/2070425963 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940043630 con 0x7f394810c510 2024-04-23T18:24:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.604+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39481a51d0 con 0x7f394810c510 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f393f7fe640 1 -- 172.21.15.26:0/2070425963 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3940036070 con 0x7f394810c510 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 Processor -- start 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 -- start start 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7db0 0x5597329c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55973291c6d0 con 0x5597329b7490 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5597328b91a0 con 0x5597329c2480 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f39533e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x559732918690 con 0x5597329b7db0 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f394dc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f394dc6a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54954/0 (socket says 172.21.15.26:54954) 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f394dc6a640 1 -- 172.21.15.26:0/1098406670 learned_addr learned my addr 172.21.15.26:0/1098406670 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.607+0000 7f394e46b640 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7db0 0x5597329c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f393f7fe640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 0x7f39340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 -- 172.21.15.26:0/1098406670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7db0 msgr2=0x5597329c1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7db0 0x5597329c1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 -- 172.21.15.26:0/1098406670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 msgr2=0x5597329c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 -- 172.21.15.26:0/1098406670 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597328c35b0 con 0x5597329b7490 2024-04-23T18:24:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394ec6c640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 0x7f39340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f393f7fe640 1 -- 172.21.15.26:0/2070425963 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f39400f1490 con 0x7f394810c510 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3938013530 tx=0x7f393800a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d174dc515894d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f393cff9640 1 -- 172.21.15.26:0/1098406670 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938019070 con 0x5597329b7490 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f393cff9640 1 -- 172.21.15.26:0/1098406670 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3938011d60 con 0x5597329b7490 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.608+0000 7f393cff9640 1 -- 172.21.15.26:0/1098406670 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393801ecf0 con 0x5597329b7490 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- 172.21.15.26:0/1098406670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 msgr2=0x5597329b7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f394ec6c640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 0x7f39340a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3940049000 tx=0x7f39400095c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3938013530 tx=0x7f393800a780 comp rx=0 tx=0).stop 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- 172.21.15.26:0/1098406670 shutdown_connections 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7db0 0x5597329c1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 --2- 172.21.15.26:0/1098406670 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7490 0x5597329b7870 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- 172.21.15.26:0/1098406670 >> 172.21.15.26:0/1098406670 conn(0x5597329b37b0 msgr2=0x5597329b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- 172.21.15.26:0/1098406670 shutdown_connections 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- 172.21.15.26:0/1098406670 wait complete. 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 -- start start 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.609+0000 7f39533e7740 1 Processor -- start 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 0x5597329c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329c2480 0x5597329bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5597329c5310 con 0x5597329b7db0 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5597328b6b20 con 0x5597329b7490 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5597329c5490 con 0x5597329c2480 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 10 client.0 _ll_register_callbacks cb 0x5597328b6530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54966/0 (socket says 172.21.15.26:54966) 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 -- 172.21.15.26:0/2165730660 learned_addr learned my addr 172.21.15.26:0/2165730660 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329c2480 0x5597329bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 0x5597329c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 -- 172.21.15.26:0/2165730660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329c2480 msgr2=0x5597329bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329c2480 0x5597329bf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 -- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 msgr2=0x5597329c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 0x5597329c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 -- 172.21.15.26:0/2165730660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55973262e920 con 0x5597329b7db0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 0x5597329c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f394e46b640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3944012740 tx=0x7f3944012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4294a2f34ebdbb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f3931ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394400f920 con 0x5597329b7db0 2024-04-23T18:24:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f394400fac0 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3938011d60 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.610+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f394400fd00 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 10 client.16485 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5597326fad30 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3938011d60 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f394401c420 con 0x5597329b7db0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 10 client.16485 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 10 client.16485 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 20 client.16485 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597329d27f0 0x5597329d4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 4 client.16485 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597328da500 con 0x5597329d27f0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.611+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597329d27f0 0x5597329d4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.612+0000 7f394dc6a640 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597329d27f0 0x5597329d4c60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebf90018a10a54c4 server_cookie=7382d01b544ff92e in_seq=0 out_seq=0 2024-04-23T18:24:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.612+0000 7f3931ffb640 10 client.16485 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.612+0000 7f3931ffb640 1 -- 172.21.15.26:0/2165730660 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597328da500 con 0x5597329d27f0 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.612+0000 7f3931ffb640 10 client.16485 handle_command_reply: tid=0 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.612+0000 7f39533e7740 1 client.16485 shutdown 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 msgr2=0x5597329bf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3944012740 tx=0x7f3944012c00 comp rx=0 tx=0).stop 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 shutdown_connections 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5597329d27f0 0x5597329d4c60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329c2480 0x5597329bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329bf200 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f3931ffb640 0 client.16485 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 --2- 172.21.15.26:0/2165730660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329b7490 0x5597329c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 >> 172.21.15.26:0/2165730660 conn(0x5597329b37b0 msgr2=0x5597329b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 shutdown_connections 2024-04-23T18:24:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 1 -- 172.21.15.26:0/2165730660 wait complete. 2024-04-23T18:24:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.613+0000 7f39533e7740 20 client.16485 trim_cache size 0 max 16384 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 Processor -- start 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 -- start start 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f39481b03c0 con 0x5597329b7db0 2024-04-23T18:24:01.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f39481adbd0 con 0x5597329c2480 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f39533e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f39481aba20 con 0x5597329b7490 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54974/0 (socket says 172.21.15.26:54974) 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 -- 172.21.15.26:0/1351962314 learned_addr learned my addr 172.21.15.26:0/1351962314 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 -- 172.21.15.26:0/1351962314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 msgr2=0x5597329ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 -- 172.21.15.26:0/1351962314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 msgr2=0x5597329da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.630+0000 7f394e46b640 1 -- 172.21.15.26:0/1351962314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597329be1d0 con 0x5597329b7db0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f394e46b640 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f394401c8e0 tx=0x7f3944005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90c3ccf00aaeff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f3933fff640 1 -- 172.21.15.26:0/1351962314 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944029070 con 0x5597329b7db0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f3933fff640 1 -- 172.21.15.26:0/1351962314 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3944012e50 con 0x5597329b7db0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f3933fff640 1 -- 172.21.15.26:0/1351962314 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440035f0 con 0x5597329b7db0 2024-04-23T18:24:01.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 -- 172.21.15.26:0/1351962314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 msgr2=0x5597329b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f394401c8e0 tx=0x7f3944005820 comp rx=0 tx=0).stop 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 -- 172.21.15.26:0/1351962314 shutdown_connections 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329b29d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 --2- 172.21.15.26:0/1351962314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 -- 172.21.15.26:0/1351962314 >> 172.21.15.26:0/1351962314 conn(0x5597329b37b0 msgr2=0x5597329b8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 -- 172.21.15.26:0/1351962314 shutdown_connections 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.631+0000 7f39533e7740 1 -- 172.21.15.26:0/1351962314 wait complete. 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 -- start start 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 Processor -- start 2024-04-23T18:24:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329c5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f39481ac090 con 0x5597329b7db0 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f39481ad500 con 0x5597329c2480 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f39481ac760 con 0x5597329b7490 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f39533e7740 10 client.0 _ll_register_callbacks cb 0x7f3944009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:54976/0 (socket says 172.21.15.26:54976) 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 -- 172.21.15.26:0/1189392704 learned_addr learned my addr 172.21.15.26:0/1189392704 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 -- 172.21.15.26:0/1189392704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 msgr2=0x5597329c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 -- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 msgr2=0x5597329c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 -- 172.21.15.26:0/1189392704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559732a38fe0 con 0x5597329b7db0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f394e46b640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3944009ef0 tx=0x7f394401c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89da5a79156c6aec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.632+0000 7f3931ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944031070 con 0x5597329b7db0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f394401cc60 con 0x5597329b7db0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3944012e50 con 0x5597329b7db0 2024-04-23T18:24:01.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3944029070 con 0x5597329b7db0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 10 client.16491 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559732a3a300 con 0x5597329b7db0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3944012e50 con 0x5597329b7db0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3944004b40 con 0x5597329b7db0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 10 client.16491 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 10 client.16491 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 20 client.16491 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559732a3a9a0 0x559732a3cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 4 client.16491 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "27958699", "release": true, "await": true}] 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "27958699", "release": true, "await": true}) -- 0x55973292a0f0 con 0x559732a3a9a0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.633+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559732a3a9a0 0x559732a3cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.634+0000 7f394dc6a640 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559732a3a9a0 0x559732a3cdc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e49e23f9059266 server_cookie=2a706e854d0eb046 in_seq=0 out_seq=0 2024-04-23T18:24:01.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.634+0000 7f3931ffb640 10 client.16491 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f3931ffb640 1 -- 172.21.15.26:0/1189392704 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55973292a0f0 con 0x559732a3a9a0 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f3931ffb640 10 client.16491 handle_command_reply: tid=0 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 client.16491 shutdown 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 msgr2=0x5597329c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3944009ef0 tx=0x7f394401c650 comp rx=0 tx=0).stop 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 shutdown_connections 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559732a3a9a0 0x559732a3cdc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5597329c2480 0x5597329c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5597329b7db0 0x5597329c5c40 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f3931ffb640 0 client.16491 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 --2- 172.21.15.26:0/1189392704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5597329b7490 0x5597329c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 >> 172.21.15.26:0/1189392704 conn(0x5597329b37b0 msgr2=0x5597329b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 shutdown_connections 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 1 -- 172.21.15.26:0/1189392704 wait complete. 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39533e7740 20 client.16491 trim_cache size 0 max 16384 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 37, 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:24:01.636 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 32, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "27958699": { 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 32, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 70.0, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:24:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 msgr2=0x7f39340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 0x7f39340a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3940049000 tx=0x7f39400095c0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 msgr2=0x7f394819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.638+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3940006010 tx=0x7f394002fbe0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 shutdown_connections 2024-04-23T18:24:01.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f39340a3d20 0x7f39340a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f394810c510 0x7f394819cf20 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3948103af0 0x7f3948198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 --2- 172.21.15.26:0/2070425963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39481031d0 0x7f3948198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 >> 172.21.15.26:0/2070425963 conn(0x7f3948077f50 msgr2=0x7f39480fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 shutdown_connections 2024-04-23T18:24:01.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:01.639+0000 7f39506cf640 1 -- 172.21.15.26:0/2070425963 wait complete. 2024-04-23T18:24:01.642 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '27958699', seconds elapsed: 11.1 2024-04-23T18:24:01.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 408186 5.75 MB/sec execute 236 sec latency 10541.289 ms 2024-04-23T18:24:02.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:02 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4159715038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:02.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:02 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4159715038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:02.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:02 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4159715038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:02.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 409526 5.75 MB/sec execute 237 sec latency 15.785 ms 2024-04-23T18:24:03.359 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:03 smithi026 ceph-mon[29364]: pgmap v210: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-04-23T18:24:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:03 smithi110 ceph-mon[32585]: pgmap v210: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-04-23T18:24:03.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:03 smithi131 ceph-mon[33041]: pgmap v210: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-04-23T18:24:03.679 INFO:tasks.workunit.client.0.smithi026.stdout: 1 411766 5.78 MB/sec execute 238 sec latency 22.567 ms 2024-04-23T18:24:04.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 412985 5.78 MB/sec execute 239 sec latency 24.397 ms 2024-04-23T18:24:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:05 smithi110 ceph-mon[32585]: pgmap v211: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-04-23T18:24:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:05 smithi026 ceph-mon[29364]: pgmap v211: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-04-23T18:24:05.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:05 smithi131 ceph-mon[33041]: pgmap v211: 97 pgs: 97 active+clean; 234 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-04-23T18:24:05.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 415176 5.81 MB/sec execute 240 sec latency 58.352 ms 2024-04-23T18:24:06.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 416144 5.80 MB/sec execute 241 sec latency 47.257 ms 2024-04-23T18:24:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:07 smithi110 ceph-mon[32585]: pgmap v212: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T18:24:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:07 smithi110 ceph-mon[32585]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:24:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:07 smithi026 ceph-mon[29364]: pgmap v212: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T18:24:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:07 smithi026 ceph-mon[29364]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:24:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:07 smithi131 ceph-mon[33041]: pgmap v212: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T18:24:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:07 smithi131 ceph-mon[33041]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:24:07.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 418262 5.82 MB/sec execute 242 sec latency 17.363 ms 2024-04-23T18:24:08.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 419642 5.83 MB/sec execute 243 sec latency 11.978 ms 2024-04-23T18:24:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:09 smithi110 ceph-mon[32585]: pgmap v214: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:09 smithi026 ceph-mon[29364]: pgmap v214: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:09.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:09 smithi131 ceph-mon[33041]: pgmap v214: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- 172.21.15.26:0/3700742272 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe8981031d0 msgr2=0x7fe8981035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3700742272 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe8981031d0 0x7fe8981035b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe888009900 tx=0x7fe88802a480 comp rx=0 tx=0).stop 2024-04-23T18:24:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- 172.21.15.26:0/3700742272 shutdown_connections 2024-04-23T18:24:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3700742272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe89810c5a0 0x7fe89810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3700742272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898103af0 0x7fe89810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3700742272 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe8981031d0 0x7fe8981035b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- 172.21.15.26:0/3700742272 >> 172.21.15.26:0/3700742272 conn(0x7fe898077fa0 msgr2=0x7fe8980783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- 172.21.15.26:0/3700742272 shutdown_connections 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- 172.21.15.26:0/3700742272 wait complete. 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 Processor -- start 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.616+0000 7fe89d94f640 1 -- start start 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8981031d0 0x7fe8981a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 0x7fe8981a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 0x7fe89819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe898112200 con 0x7fe89810c5a0 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe898112080 con 0x7fe898103af0 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe898112380 con 0x7fe8981031d0 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 0x7fe8981a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 0x7fe89819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 0x7fe8981a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:39226/0 (socket says 172.21.15.26:39226) 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe88ffff640 1 -- 172.21.15.26:0/3711013734 learned_addr learned my addr 172.21.15.26:0/3711013734 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe896ffd640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8981031d0 0x7fe8981a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 -- 172.21.15.26:0/3711013734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8981031d0 msgr2=0x7fe8981a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8981031d0 0x7fe8981a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 -- 172.21.15.26:0/3711013734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 msgr2=0x7fe8981a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 0x7fe8981a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 -- 172.21.15.26:0/3711013734 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89819b030 con 0x7fe89810c5a0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe8977fe640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 0x7fe89819a770 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe890009fd0 tx=0x7fe89000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89c94d640 1 -- 172.21.15.26:0/3711013734 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89000ce20 con 0x7fe89810c5a0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89819b320 con 0x7fe89810c5a0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89c94d640 1 -- 172.21.15.26:0/3711013734 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe890005020 con 0x7fe89810c5a0 2024-04-23T18:24:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.617+0000 7fe89c94d640 1 -- 172.21.15.26:0/3711013734 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890010690 con 0x7fe89810c5a0 2024-04-23T18:24:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.618+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8981a75e0 con 0x7fe89810c5a0 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.620+0000 7fe89c94d640 1 -- 172.21.15.26:0/3711013734 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe8900054d0 con 0x7fe89810c5a0 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 Processor -- start 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 -- start start 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc78800 0x55d7fbc7abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbbd2970 con 0x55d7fbc6d620 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbb6f4a0 con 0x55d7fbc6df40 2024-04-23T18:24:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe8a066e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbbce930 con 0x55d7fbc78800 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49474/0 (socket says 172.21.15.26:49474) 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe88ffff640 1 -- 172.21.15.26:0/4010694540 learned_addr learned my addr 172.21.15.26:0/4010694540 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe88ffff640 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc78800 0x55d7fbc7abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe89c94d640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 0x7fe8840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe896ffd640 1 -- 172.21.15.26:0/4010694540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc78800 msgr2=0x55d7fbc7abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.621+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc78800 0x55d7fbc7abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe896ffd640 1 -- 172.21.15.26:0/4010694540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 msgr2=0x55d7fbc6da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe8977fe640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 0x7fe8840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe896ffd640 1 -- 172.21.15.26:0/4010694540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7fb8e5210 con 0x55d7fbc6df40 2024-04-23T18:24:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe88ffff640 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe89c94d640 1 -- 172.21.15.26:0/3711013734 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7fe8900d0ea0 con 0x7fe89810c5a0 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc782c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe88802aa70 tx=0x7fe888004190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d50bc0f299e662 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe8977fe640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 0x7fe8840a6280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe89009cb20 tx=0x7fe89001fde0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe88dffb640 1 -- 172.21.15.26:0/4010694540 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888035040 con 0x55d7fbc6df40 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe88dffb640 1 -- 172.21.15.26:0/4010694540 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8880054f0 con 0x55d7fbc6df40 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.622+0000 7fe88dffb640 1 -- 172.21.15.26:0/4010694540 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888004d90 con 0x55d7fbc6df40 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- 172.21.15.26:0/4010694540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 msgr2=0x55d7fbc782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc782c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe88802aa70 tx=0x7fe888004190 comp rx=0 tx=0).stop 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- 172.21.15.26:0/4010694540 shutdown_connections 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc782c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc78800 0x55d7fbc7abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4010694540 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6d620 0x55d7fbc6da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- 172.21.15.26:0/4010694540 >> 172.21.15.26:0/4010694540 conn(0x55d7fbc69b30 msgr2=0x55d7fbc69f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- 172.21.15.26:0/4010694540 shutdown_connections 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- 172.21.15.26:0/4010694540 wait complete. 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 -- start start 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.623+0000 7fe8a066e740 1 Processor -- start 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc78110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc78800 0x55d7fbc75940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbc7b3d0 con 0x55d7fbc78800 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbb6d0b0 con 0x55d7fbc6df40 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d7fbc7b550 con 0x55d7fbc6d620 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe8a066e740 10 client.0 _ll_register_callbacks cb 0x55d7fbb5ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc78110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc78110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:49310/0 (socket says 172.21.15.26:49310) 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe88ffff640 1 -- 172.21.15.26:0/1916919860 learned_addr learned my addr 172.21.15.26:0/1916919860 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:39240/0 (socket says 172.21.15.26:39240) 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe88ffff640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc78800 0x55d7fbc75940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 -- 172.21.15.26:0/1916919860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 msgr2=0x55d7fbc78110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc78110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 -- 172.21.15.26:0/1916919860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc78800 msgr2=0x55d7fbc75940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc78800 0x55d7fbc75940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 -- 172.21.15.26:0/1916919860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7fb9b1620 con 0x55d7fbc6df40 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe896ffd640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe888005ab0 tx=0x7fe888005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d72093fddb3d2f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe876ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.624+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888053060 con 0x55d7fbc6df40 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8880054f0 con 0x55d7fbc6df40 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7fe888004cf0 con 0x55d7fbc6df40 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe888035050 con 0x55d7fbc6df40 2024-04-23T18:24:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 10 client.25901 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7fbc872f0 con 0x55d7fbc6df40 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8880054f0 con 0x55d7fbc6df40 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe888003a70 con 0x55d7fbc6df40 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 10 client.25901 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 10 client.25901 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 20 client.25901 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbc88640 0x55d7fbc8aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 4 client.25901 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7fbbe0390 con 0x55d7fbc88640 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.625+0000 7fe88ffff640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbc88640 0x55d7fbc8aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.626+0000 7fe876ffd640 10 client.25901 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.626+0000 7fe88ffff640 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbc88640 0x55d7fbc8aa30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec2c5c0af806500 server_cookie=425c20a3535622c8 in_seq=0 out_seq=0 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.626+0000 7fe876ffd640 1 -- 172.21.15.26:0/1916919860 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7fbbe0390 con 0x55d7fbc88640 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.626+0000 7fe876ffd640 10 client.25901 handle_command_reply: tid=0 2024-04-23T18:24:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.626+0000 7fe8a066e740 1 client.25901 shutdown 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 msgr2=0x55d7fbc75400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe888005ab0 tx=0x7fe888005710 comp rx=0 tx=0).stop 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 shutdown_connections 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbc88640 0x55d7fbc8aa30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc6df40 0x55d7fbc75400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe876ffd640 0 client.25901 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc78800 0x55d7fbc75940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 --2- 172.21.15.26:0/1916919860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc78110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 >> 172.21.15.26:0/1916919860 conn(0x55d7fbc69b30 msgr2=0x55d7fbc6f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 shutdown_connections 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 1 -- 172.21.15.26:0/1916919860 wait complete. 2024-04-23T18:24:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.627+0000 7fe8a066e740 20 client.25901 trim_cache size 0 max 16384 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 Processor -- start 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 -- start start 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc84290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc6f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981b2680 con 0x55d7fbc6df40 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981afe90 con 0x55d7fbc78800 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe8a066e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981add00 con 0x55d7fbc6d620 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc84290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49490/0 (socket says 172.21.15.26:49490) 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc84290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:49324/0 (socket says 172.21.15.26:49324) 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 -- 172.21.15.26:0/4040071220 learned_addr learned my addr 172.21.15.26:0/4040071220 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe88ffff640 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc6f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 -- 172.21.15.26:0/4040071220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 msgr2=0x55d7fbc84290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc84290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 -- 172.21.15.26:0/4040071220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 msgr2=0x55d7fbc6f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc6f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 -- 172.21.15.26:0/4040071220 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7fbc73950 con 0x55d7fbc6df40 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.644+0000 7fe896ffd640 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe888031440 tx=0x7fe888004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf255017f8e0eee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe88d7fa640 1 -- 172.21.15.26:0/4040071220 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888053040 con 0x55d7fbc6df40 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe88d7fa640 1 -- 172.21.15.26:0/4040071220 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88804a070 con 0x55d7fbc6df40 2024-04-23T18:24:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe88d7fa640 1 -- 172.21.15.26:0/4040071220 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8880039f0 con 0x55d7fbc6df40 2024-04-23T18:24:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 -- 172.21.15.26:0/4040071220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 msgr2=0x55d7fbc77c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe888031440 tx=0x7fe888004590 comp rx=0 tx=0).stop 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 -- 172.21.15.26:0/4040071220 shutdown_connections 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc6f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc77c40 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 --2- 172.21.15.26:0/4040071220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc84290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.645+0000 7fe8a066e740 1 -- 172.21.15.26:0/4040071220 >> 172.21.15.26:0/4040071220 conn(0x55d7fbc69b30 msgr2=0x55d7fbc6efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- 172.21.15.26:0/4040071220 shutdown_connections 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- 172.21.15.26:0/4040071220 wait complete. 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- start start 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 Processor -- start 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981ae350 con 0x55d7fbc6df40 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981af7c0 con 0x55d7fbc78800 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981aea20 con 0x55d7fbc6d620 2024-04-23T18:24:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.646+0000 7fe8a066e740 10 client.0 _ll_register_callbacks cb 0x7fe888005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49496/0 (socket says 172.21.15.26:49496) 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 -- 172.21.15.26:0/3462097984 learned_addr learned my addr 172.21.15.26:0/3462097984 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:49336/0 (socket says 172.21.15.26:49336) 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe88ffff640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 -- 172.21.15.26:0/3462097984 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 msgr2=0x55d7fbc80a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 -- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 msgr2=0x55d7fbc721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 -- 172.21.15.26:0/3462097984 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7fbc728b0 con 0x55d7fbc6df40 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe88ffff640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe896ffd640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe88802a4c0 tx=0x7fe888005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfaaab463cfd67ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe876ffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888053070 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe88804a070 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.647+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7fe888003980 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe88806f050 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 10 client.16509 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7fbcee810 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88804a070 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe8880576b0 con 0x55d7fbc6df40 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 10 client.16509 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 10 client.16509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 20 client.16509 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbceed20 0x55d7fbcf1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 4 client.16509 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7fbbda200 con 0x55d7fbceed20 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.648+0000 7fe88ffff640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbceed20 0x55d7fbcf1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.649+0000 7fe88ffff640 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbceed20 0x55d7fbcf1140 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4c251ba3000d604 server_cookie=50e3e89401e7570 in_seq=0 out_seq=0 2024-04-23T18:24:09.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.649+0000 7fe876ffd640 10 client.16509 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe876ffd640 1 -- 172.21.15.26:0/3462097984 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7fbbda200 con 0x55d7fbceed20 2024-04-23T18:24:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe876ffd640 10 client.16509 handle_command_reply: tid=0 2024-04-23T18:24:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 client.16509 shutdown 2024-04-23T18:24:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 msgr2=0x55d7fbc71cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe88802a4c0 tx=0x7fe888005a20 comp rx=0 tx=0).stop 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 shutdown_connections 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d7fbceed20 0x55d7fbcf1140 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d7fbc78800 0x55d7fbc721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d7fbc6df40 0x55d7fbc71cb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe876ffd640 0 client.16509 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 --2- 172.21.15.26:0/3462097984 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7fbc6d620 0x55d7fbc80a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 >> 172.21.15.26:0/3462097984 conn(0x55d7fbc69b30 msgr2=0x55d7fbc6f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 shutdown_connections 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 1 -- 172.21.15.26:0/3462097984 wait complete. 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.679+0000 7fe8a066e740 20 client.16509 trim_cache size 0 max 16384 2024-04-23T18:24:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 msgr2=0x7fe8840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 0x7fe8840a6280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe89009cb20 tx=0x7fe89001fde0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 msgr2=0x7fe89819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 0x7fe89819a770 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe890009fd0 tx=0x7fe89000ef10 comp rx=0 tx=0).stop 2024-04-23T18:24:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 shutdown_connections 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe8840a3dc0 0x7fe8840a6280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe898103af0 0x7fe8981a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe89810c5a0 0x7fe89819a770 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 --2- 172.21.15.26:0/3711013734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8981031d0 0x7fe8981a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 >> 172.21.15.26:0/3711013734 conn(0x7fe898077fa0 msgr2=0x7fe898108420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 shutdown_connections 2024-04-23T18:24:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:09.680+0000 7fe89d94f640 1 -- 172.21.15.26:0/3711013734 wait complete. 2024-04-23T18:24:09.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 421749 5.85 MB/sec execute 244 sec latency 17.565 ms 2024-04-23T18:24:10.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 422958 5.86 MB/sec execute 245 sec latency 38.042 ms 2024-04-23T18:24:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:11 smithi110 ceph-mon[32585]: pgmap v215: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:11 smithi026 ceph-mon[29364]: pgmap v215: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:11.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:11 smithi131 ceph-mon[33041]: pgmap v215: 97 pgs: 97 active+clean; 232 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 188 op/s 2024-04-23T18:24:11.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 425144 5.88 MB/sec execute 246 sec latency 21.176 ms 2024-04-23T18:24:12.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 426560 5.89 MB/sec execute 247 sec latency 22.936 ms 2024-04-23T18:24:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:13 smithi110 ceph-mon[32585]: pgmap v216: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:13.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:13 smithi026 ceph-mon[29364]: pgmap v216: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:13 smithi131 ceph-mon[33041]: pgmap v216: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:13.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 428105 5.89 MB/sec execute 248 sec latency 14.878 ms 2024-04-23T18:24:14.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 429818 5.91 MB/sec execute 249 sec latency 25.171 ms 2024-04-23T18:24:15.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:15 smithi110 ceph-mon[32585]: pgmap v217: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:15.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:15 smithi026 ceph-mon[29364]: pgmap v217: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:15.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:15 smithi131 ceph-mon[33041]: pgmap v217: 97 pgs: 97 active+clean; 244 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 279 op/s 2024-04-23T18:24:15.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 431639 5.92 MB/sec execute 250 sec latency 58.730 ms 2024-04-23T18:24:16.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 433296 5.94 MB/sec execute 251 sec latency 14.354 ms 2024-04-23T18:24:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:17 smithi110 ceph-mon[32585]: pgmap v218: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:17 smithi110 ceph-mon[32585]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:24:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:17 smithi026 ceph-mon[29364]: pgmap v218: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:17 smithi026 ceph-mon[29364]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:24:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:17 smithi131 ceph-mon[33041]: pgmap v218: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:17 smithi131 ceph-mon[33041]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:24:17.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 435173 5.95 MB/sec execute 252 sec latency 20.424 ms 2024-04-23T18:24:18.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 436792 5.97 MB/sec execute 253 sec latency 15.611 ms 2024-04-23T18:24:19.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:19 smithi110 ceph-mon[32585]: pgmap v220: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:19 smithi026 ceph-mon[29364]: pgmap v220: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:19.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:19 smithi131 ceph-mon[33041]: pgmap v220: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:19.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 438376 5.97 MB/sec execute 254 sec latency 22.651 ms 2024-04-23T18:24:20.214 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 35.0 2024-04-23T18:24:20.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:20.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 -- 172.21.15.26:0/448084462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f115810c520 msgr2=0x7f115810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 --2- 172.21.15.26:0/448084462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f115810c520 0x7f115810e9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1150009ca0 tx=0x7f115002a9e0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 -- 172.21.15.26:0/448084462 shutdown_connections 2024-04-23T18:24:20.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 --2- 172.21.15.26:0/448084462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1158103b00 0x7f115810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 --2- 172.21.15.26:0/448084462 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f115810c520 0x7f115810e9c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 --2- 172.21.15.26:0/448084462 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11581031e0 0x7f11581035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 -- 172.21.15.26:0/448084462 >> 172.21.15.26:0/448084462 conn(0x7f1158077f50 msgr2=0x7f1158078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 -- 172.21.15.26:0/448084462 shutdown_connections 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.330+0000 7f115ef68640 1 -- 172.21.15.26:0/448084462 wait complete. 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 Processor -- start 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 -- start start 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 0x7f1158100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 0x7f11581018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f11581121b0 con 0x7f1158103b00 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1158112030 con 0x7f11581031e0 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115ef68640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f1158112330 con 0x7f115810c520 2024-04-23T18:24:20.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f1157fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f1157fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44036/0 (socket says 172.21.15.26:44036) 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f1157fff640 1 -- 172.21.15.26:0/3798656940 learned_addr learned my addr 172.21.15.26:0/3798656940 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115cd04640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 0x7f1158100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.331+0000 7f115d505640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 0x7f11581018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 -- 172.21.15.26:0/3798656940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 msgr2=0x7f11581018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 0x7f11581018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 -- 172.21.15.26:0/3798656940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 msgr2=0x7f1158100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 0x7f1158100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 -- 172.21.15.26:0/3798656940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581a6bd0 con 0x7f1158103b00 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f115d505640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 0x7f11581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f115cd04640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 0x7f1158100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1157fff640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f114000e9f0 tx=0x7f114000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114000cd80 con 0x7f1158103b00 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11581a6ec0 con 0x7f1158103b00 2024-04-23T18:24:20.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11400108e0 con 0x7f1158103b00 2024-04-23T18:24:20.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140018980 con 0x7f1158103b00 2024-04-23T18:24:20.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.332+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581a73d0 con 0x7f1158103b00 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.333+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1158068d70 con 0x7f1158103b00 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.334+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1140010430 con 0x7f1158103b00 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.335+0000 7f1155ffb640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 0x7f11440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.335+0000 7f115cd04640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 0x7f11440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.335+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f11400d0ae0 con 0x7f1158103b00 2024-04-23T18:24:20.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.336+0000 7f115cd04640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 0x7f11440a6210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1148005fd0 tx=0x7f114800b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.340+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f114009d180 con 0x7f1158103b00 2024-04-23T18:24:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.595+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f115802d5c0 con 0x7f1158103b00 2024-04-23T18:24:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.595+0000 7f1155ffb640 1 -- 172.21.15.26:0/3798656940 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11893 (secure 0 0 0) 0x7f114009ca80 con 0x7f1158103b00 2024-04-23T18:24:20.588 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:24:20.588 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:22:59.417042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:24:20.589 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 37 2024-04-23T18:24:20.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 msgr2=0x7f11440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 0x7f11440a6210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1148005fd0 tx=0x7f114800b390 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 msgr2=0x7f1158101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f114000e9f0 tx=0x7f114000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 shutdown_connections 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f11440a3d50 0x7f11440a6210 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f115810c520 0x7f11581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f1158103b00 0x7f1158101370 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 --2- 172.21.15.26:0/3798656940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11581031e0 0x7f1158100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 >> 172.21.15.26:0/3798656940 conn(0x7f1158077f50 msgr2=0x7f11580fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 shutdown_connections 2024-04-23T18:24:20.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.597+0000 7f115ef68640 1 -- 172.21.15.26:0/3798656940 wait complete. 2024-04-23T18:24:20.602 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 95.0 --await-for 42.0' 2024-04-23T18:24:20.602 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 95.0 --await-for 42.0 2024-04-23T18:24:20.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440258 6.00 MB/sec execute 255 sec latency 34.361 ms 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 -- 172.21.15.26:0/2536911201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f902806a820 msgr2=0x7f902806ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 --2- 172.21.15.26:0/2536911201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f902806a820 0x7f902806ac00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9018009900 tx=0x7f901802a410 comp rx=0 tx=0).stop 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 -- 172.21.15.26:0/2536911201 shutdown_connections 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 --2- 172.21.15.26:0/2536911201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806b1d0 0x7f9028106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 --2- 172.21.15.26:0/2536911201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9028107260 0x7f902810ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 --2- 172.21.15.26:0/2536911201 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f902806a820 0x7f902806ac00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.728+0000 7f903021e640 1 -- 172.21.15.26:0/2536911201 >> 172.21.15.26:0/2536911201 conn(0x7f9028076f60 msgr2=0x7f9028077370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 -- 172.21.15.26:0/2536911201 shutdown_connections 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 -- 172.21.15.26:0/2536911201 wait complete. 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 Processor -- start 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 -- start start 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.729+0000 7f903021e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f902819d020 0x7f902819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f903021e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f90281121a0 con 0x7f9028107260 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f903021e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9028112020 con 0x7f902806a820 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f903021e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f9028112320 con 0x7f902819d020 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50094/0 (socket says 172.21.15.26:50094) 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 -- 172.21.15.26:0/691722835 learned_addr learned my addr 172.21.15.26:0/691722835 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44056/0 (socket says 172.21.15.26:44056) 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902e7bb640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f902819d020 0x7f902819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 -- 172.21.15.26:0/691722835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f902819d020 msgr2=0x7f902819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f902819d020 0x7f902819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 -- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 msgr2=0x7f9028198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 -- 172.21.15.26:0/691722835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902819db10 con 0x7f902806a820 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902d7b9640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f902dfba640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9018005940 tx=0x7f9018004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.730+0000 7f901effd640 1 -- 172.21.15.26:0/691722835 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901803f030 con 0x7f902806a820 2024-04-23T18:24:20.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.731+0000 7f901effd640 1 -- 172.21.15.26:0/691722835 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9018003aa0 con 0x7f902806a820 2024-04-23T18:24:20.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.731+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902819dda0 con 0x7f902806a820 2024-04-23T18:24:20.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.731+0000 7f901effd640 1 -- 172.21.15.26:0/691722835 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018035070 con 0x7f902806a820 2024-04-23T18:24:20.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.731+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90281a5560 con 0x7f902806a820 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f901effd640 1 -- 172.21.15.26:0/691722835 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9018042030 con 0x7f902806a820 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f903300d740 1 Processor -- start 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f903300d740 1 -- start start 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f903300d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dfd91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f903300d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfe7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.734+0000 7f903300d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfea890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f903300d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d75df3e370 con 0x55d75dfe84a0 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f903300d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d75dedb2e0 con 0x55d75dfd9780 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f903300d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d75df3a330 con 0x55d75dfd8dd0 2024-04-23T18:24:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfe7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfea890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dfd91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfe7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50100/0 (socket says 172.21.15.26:50100) 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902dfba640 1 -- 172.21.15.26:0/3081983239 learned_addr learned my addr 172.21.15.26:0/3081983239 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f901effd640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 0x7f90140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 -- 172.21.15.26:0/3081983239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 msgr2=0x55d75dfe7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfe7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 -- 172.21.15.26:0/3081983239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 msgr2=0x55d75dfea890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfea890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 -- 172.21.15.26:0/3081983239 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75dd1cd80 con 0x55d75dfd8dd0 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f901effd640 1 -- 172.21.15.26:0/691722835 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f90180f0e40 con 0x7f902806a820 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 0x7f90140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.735+0000 7f902e7bb640 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dfd91b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9020003a70 tx=0x7f9020003f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd624a440a6d131b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f9013fff640 1 -- 172.21.15.26:0/3081983239 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020024070 con 0x55d75dfd8dd0 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f9013fff640 1 -- 172.21.15.26:0/3081983239 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9020004d20 con 0x55d75dfd8dd0 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f9013fff640 1 -- 172.21.15.26:0/3081983239 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020005330 con 0x55d75dfd8dd0 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 -- 172.21.15.26:0/3081983239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 msgr2=0x55d75dfd91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f902e7bb640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 0x7f90140a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9020021000 tx=0x7f902001e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dfd91b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9020003a70 tx=0x7f9020003f30 comp rx=0 tx=0).stop 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 -- 172.21.15.26:0/3081983239 shutdown_connections 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfe7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfea890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 --2- 172.21.15.26:0/3081983239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dfd91b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.736+0000 7f903300d740 1 -- 172.21.15.26:0/3081983239 >> 172.21.15.26:0/3081983239 conn(0x55d75dfd53c0 msgr2=0x55d75dfd57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- 172.21.15.26:0/3081983239 shutdown_connections 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- 172.21.15.26:0/3081983239 wait complete. 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- start start 2024-04-23T18:24:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 Processor -- start 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfe5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d75dfeb260 con 0x55d75dfe84a0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d75ded8fb0 con 0x55d75dfd8dd0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d75dfeb3e0 con 0x55d75dfd9780 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.737+0000 7f903300d740 10 client.0 _ll_register_callbacks cb 0x55d75ded89a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfe5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50102/0 (socket says 172.21.15.26:50102) 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44064/0 (socket says 172.21.15.26:44064) 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 -- 172.21.15.26:0/3706154309 learned_addr learned my addr 172.21.15.26:0/3706154309 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfe5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:36290/0 (socket says 172.21.15.26:36290) 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 -- 172.21.15.26:0/3706154309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 msgr2=0x55d75dfe5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfe5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 -- 172.21.15.26:0/3706154309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 msgr2=0x55d75dfe5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 -- 172.21.15.26:0/3706154309 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75dd7d510 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902e7bb640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f902dfba640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90180eeb90 tx=0x7f9018005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf215a2f50d086f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f90117fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018049070 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f90180b9500 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9020004d20 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f90180bcc60 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f903300d740 10 client.25925 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.738+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d75dc10670 con 0x55d75dfd8dd0 2024-04-23T18:24:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9020004d20 con 0x55d75dfd8dd0 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f90180eece0 con 0x55d75dfd8dd0 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 10 client.25925 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 10 client.25925 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 20 client.25925 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75dff86b0 0x55d75dffab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 4 client.25925 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d75df4bd90 con 0x55d75dff86b0 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.739+0000 7f902d7b9640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75dff86b0 0x55d75dffab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.740+0000 7f90117fa640 10 client.25925 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:20.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.740+0000 7f902d7b9640 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75dff86b0 0x55d75dffab40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69640dd9aa0401b8 server_cookie=a333d4a0b2cc4c7d in_seq=0 out_seq=0 2024-04-23T18:24:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.740+0000 7f90117fa640 1 -- 172.21.15.26:0/3706154309 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d75df4bd90 con 0x55d75dff86b0 2024-04-23T18:24:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.740+0000 7f90117fa640 10 client.25925 handle_command_reply: tid=0 2024-04-23T18:24:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 client.25925 shutdown 2024-04-23T18:24:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 msgr2=0x55d75dfe53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90180eeb90 tx=0x7f9018005b50 comp rx=0 tx=0).stop 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 shutdown_connections 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75dff86b0 0x55d75dffab40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75dfe5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f90117fa640 0 client.25925 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfe5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 --2- 172.21.15.26:0/3706154309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfe53e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 >> 172.21.15.26:0/3706154309 conn(0x55d75dfd53c0 msgr2=0x55d75dfdb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 shutdown_connections 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 1 -- 172.21.15.26:0/3706154309 wait complete. 2024-04-23T18:24:20.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.741+0000 7f903300d740 20 client.25925 trim_cache size 0 max 16384 2024-04-23T18:24:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.760+0000 7f903300d740 1 Processor -- start 2024-04-23T18:24:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 -- start start 2024-04-23T18:24:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dff4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfdb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f90281ae320 con 0x55d75dfe84a0 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f90281abb30 con 0x55d75dfd9780 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f903300d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f90281a99c0 con 0x55d75dfd8dd0 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44080/0 (socket says 172.21.15.26:44080) 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 -- 172.21.15.26:0/478510472 learned_addr learned my addr 172.21.15.26:0/478510472 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902d7b9640 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfdb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902dfba640 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dff4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 -- 172.21.15.26:0/478510472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 msgr2=0x55d75dff4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dff4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 -- 172.21.15.26:0/478510472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 msgr2=0x55d75dfdb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfdb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 -- 172.21.15.26:0/478510472 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75dffb1c0 con 0x55d75dfe84a0 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.761+0000 7f902e7bb640 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f902001ef20 tx=0x7f9020013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6480c9bc74777553 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f90137fe640 1 -- 172.21.15.26:0/478510472 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020024040 con 0x55d75dfe84a0 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f90137fe640 1 -- 172.21.15.26:0/478510472 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902001e540 con 0x55d75dfe84a0 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f90137fe640 1 -- 172.21.15.26:0/478510472 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020005270 con 0x55d75dfe84a0 2024-04-23T18:24:20.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 -- 172.21.15.26:0/478510472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 msgr2=0x55d75e000450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f902001ef20 tx=0x7f9020013000 comp rx=0 tx=0).stop 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 -- 172.21.15.26:0/478510472 shutdown_connections 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd9780 0x55d75dfdb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e000450 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 --2- 172.21.15.26:0/478510472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd8dd0 0x55d75dff4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.762+0000 7f903300d740 1 -- 172.21.15.26:0/478510472 >> 172.21.15.26:0/478510472 conn(0x55d75dfd53c0 msgr2=0x55d75dfda980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- 172.21.15.26:0/478510472 shutdown_connections 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- 172.21.15.26:0/478510472 wait complete. 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- start start 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 Processor -- start 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f90281aa010 con 0x55d75dfe84a0 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f90281ab460 con 0x55d75dfd8dd0 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f90281aa6c0 con 0x55d75dfd9780 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f903300d740 10 client.0 _ll_register_callbacks cb 0x7f90180f4680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.763+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902dfba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44094/0 (socket says 172.21.15.26:44094) 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902d7b9640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 -- 172.21.15.26:0/1269658520 learned_addr learned my addr 172.21.15.26:0/1269658520 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902dfba640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:50116/0 (socket says 172.21.15.26:50116) 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 -- 172.21.15.26:0/1269658520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 msgr2=0x55d75dfec9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfec9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 -- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 msgr2=0x55d75dfec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 -- 172.21.15.26:0/1269658520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75e05e200 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902dfba640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f902e7bb640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9020004780 tx=0x7f9020004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e5d124c477defa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f90117fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020037070 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f902001e540 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f902001eb00 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9020024040 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f903300d740 10 client.16533 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.764+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d75e05f380 con 0x55d75dfe84a0 2024-04-23T18:24:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902001e540 con 0x55d75dfe84a0 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9020029440 con 0x55d75dfe84a0 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 10 client.16533 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 10 client.16533 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 20 client.16533 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75e05f980 0x55d75e061da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 4 client.16533 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 95.0, "await_for": 42.0}] 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 95.0, "await_for": 42.0}) -- 0x55d75df45c00 con 0x55d75e05f980 2024-04-23T18:24:20.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f902dfba640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75e05f980 0x55d75e061da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:20.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f90117fa640 10 client.16533 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:20.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.765+0000 7f902dfba640 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75e05f980 0x55d75e061da0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6da8842cee8c72f server_cookie=16b655685e6982a8 in_seq=0 out_seq=0 2024-04-23T18:24:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f90117fa640 1 -- 172.21.15.26:0/1269658520 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d75df45c00 con 0x55d75e05f980 2024-04-23T18:24:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f90117fa640 10 client.16533 handle_command_reply: tid=0 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 client.16533 shutdown 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 msgr2=0x55d75e05dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9020004780 tx=0x7f9020004cb0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 shutdown_connections 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d75e05f980 0x55d75e061da0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d75dfe84a0 0x55d75e05dcc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f90117fa640 0 client.16533 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d75dfd9780 0x55d75dfec9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 --2- 172.21.15.26:0/1269658520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d75dfd8dd0 0x55d75dfec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 >> 172.21.15.26:0/1269658520 conn(0x55d75dfd53c0 msgr2=0x55d75dfdb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 shutdown_connections 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 1 -- 172.21.15.26:0/1269658520 wait complete. 2024-04-23T18:24:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.862+0000 7f903300d740 20 client.16533 trim_cache size 0 max 16384 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 37, 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 34, 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "1e98f5fd": { 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 34, 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 95.0, 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:20.859 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:24:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 msgr2=0x7f90140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 0x7f90140a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9020021000 tx=0x7f902001e000 comp rx=0 tx=0).stop 2024-04-23T18:24:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 msgr2=0x7f9028198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9018005940 tx=0x7f9018004270 comp rx=0 tx=0).stop 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 shutdown_connections 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f90140a3d20 0x7f90140a61e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f902819d020 0x7f902819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9028107260 0x7f9028198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 --2- 172.21.15.26:0/691722835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902806a820 0x7f9028198760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 >> 172.21.15.26:0/691722835 conn(0x7f9028076f60 msgr2=0x7f90280ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 shutdown_connections 2024-04-23T18:24:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:20.863+0000 7f903021e640 1 -- 172.21.15.26:0/691722835 wait complete. 2024-04-23T18:24:20.868 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1e98f5fd', quiesce took 0.7 seconds. Will release after: 35.0 2024-04-23T18:24:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:21 smithi026 ceph-mon[29364]: pgmap v221: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:21 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3798656940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:21 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:21 smithi110 ceph-mon[32585]: pgmap v221: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:21 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3798656940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:21 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:21 smithi131 ceph-mon[33041]: pgmap v221: 97 pgs: 97 active+clean; 256 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 279 op/s 2024-04-23T18:24:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:21 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3798656940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:21 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:21.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.98 MB/sec execute 256 sec latency 878.551 ms 2024-04-23T18:24:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:22 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:24:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:22 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:24:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:22 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:24:22.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.95 MB/sec execute 257 sec latency 1878.696 ms 2024-04-23T18:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.779+0000 7f54ef1fe640 1 -- 172.21.15.26:0/3760858493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e8102b70 msgr2=0x7f54e8106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.779+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/3760858493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e8102b70 0x7f54e8106bb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009900 tx=0x7f54d802a410 comp rx=0 tx=0).stop 2024-04-23T18:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.779+0000 7f54ef1fe640 1 -- 172.21.15.26:0/3760858493 shutdown_connections 2024-04-23T18:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.779+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/3760858493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.779+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/3760858493 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e8102b70 0x7f54e8106bb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/3760858493 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e81021c0 0x7f54e81025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- 172.21.15.26:0/3760858493 >> 172.21.15.26:0/3760858493 conn(0x7f54e80fda20 msgr2=0x7f54e80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- 172.21.15.26:0/3760858493 shutdown_connections 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- 172.21.15.26:0/3760858493 wait complete. 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 Processor -- start 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- start start 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e819cf30 0x7f54e819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81121b0 con 0x7f54e81021c0 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54e8112030 con 0x7f54e8107290 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.780+0000 7f54ef1fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f54e8112330 con 0x7f54e819cf30 2024-04-23T18:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44114/0 (socket says 172.21.15.26:44114) 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 -- 172.21.15.26:0/667375269 learned_addr learned my addr 172.21.15.26:0/667375269 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54dffff640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ed79b640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e819cf30 0x7f54e819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 -- 172.21.15.26:0/667375269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e819cf30 msgr2=0x7f54e819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e819cf30 0x7f54e819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 -- 172.21.15.26:0/667375269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 msgr2=0x7f54e8198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e8198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 -- 172.21.15.26:0/667375269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e819da20 con 0x7f54e81021c0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54dffff640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c920 tx=0x7f54e400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ddffb640 1 -- 172.21.15.26:0/667375269 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4007c70 con 0x7f54e81021c0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ddffb640 1 -- 172.21.15.26:0/667375269 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54e4007e10 con 0x7f54e81021c0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e819dd10 con 0x7f54e81021c0 2024-04-23T18:24:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.781+0000 7f54ddffb640 1 -- 172.21.15.26:0/667375269 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4018820 con 0x7f54e81021c0 2024-04-23T18:24:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.782+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e81a54d0 con 0x7f54e81021c0 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.784+0000 7f54ddffb640 1 -- 172.21.15.26:0/667375269 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f54e400f450 con 0x7f54e81021c0 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.784+0000 7f54f1ff2740 1 Processor -- start 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.784+0000 7f54ddffb640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 0x7f54d00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.784+0000 7f54f1ff2740 1 -- start start 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb627a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ddffb640 1 -- 172.21.15.26:0/667375269 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f54e40d0a30 con 0x7f54e81021c0 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 0x7f54d00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb6322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb58c970 con 0x55d6eb632800 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb5294a0 con 0x55d6eb627620 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb588930 con 0x55d6eb627f40 2024-04-23T18:24:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb6322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44128/0 (socket says 172.21.15.26:44128) 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 -- 172.21.15.26:0/2219459367 learned_addr learned my addr 172.21.15.26:0/2219459367 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb627a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 0x7f54d00a6280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013770 tx=0x7f54e0013b70 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 -- 172.21.15.26:0/2219459367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 msgr2=0x55d6eb627a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb627a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 -- 172.21.15.26:0/2219459367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 msgr2=0x55d6eb634bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.785 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54dffff640 1 -- 172.21.15.26:0/2219459367 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6eb29f210 con 0x55d6eb627f40 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.785+0000 7f54ed79b640 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54dffff640 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb6322c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f54d8006010 tx=0x7f54d8032e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b9c7f2f908e305 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54ceffd640 1 -- 172.21.15.26:0/2219459367 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d803f070 con 0x55d6eb627f40 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54ceffd640 1 -- 172.21.15.26:0/2219459367 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d80316a0 con 0x55d6eb627f40 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54ceffd640 1 -- 172.21.15.26:0/2219459367 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8035070 con 0x55d6eb627f40 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54f1ff2740 1 -- 172.21.15.26:0/2219459367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 msgr2=0x55d6eb6322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.786+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb6322c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f54d8006010 tx=0x7f54d8032e90 comp rx=0 tx=0).stop 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- 172.21.15.26:0/2219459367 shutdown_connections 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb634bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb6322c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/2219459367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb627a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- 172.21.15.26:0/2219459367 >> 172.21.15.26:0/2219459367 conn(0x55d6eb623b30 msgr2=0x55d6eb623f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- 172.21.15.26:0/2219459367 shutdown_connections 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- 172.21.15.26:0/2219459367 wait complete. 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- start start 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 Processor -- start 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb632290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb62f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb6353d0 con 0x55d6eb632800 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb5270b0 con 0x55d6eb627620 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.787+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55d6eb635550 con 0x55d6eb627f40 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54f1ff2740 10 client.0 _ll_register_callbacks cb 0x55d6eb514d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ecf9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb632290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54dffff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb62f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44142/0 (socket says 172.21.15.26:44142) 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 -- 172.21.15.26:0/1694710252 learned_addr learned my addr 172.21.15.26:0/1694710252 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 -- 172.21.15.26:0/1694710252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 msgr2=0x55d6eb62f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb62f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 -- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 msgr2=0x55d6eb632290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb632290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 -- 172.21.15.26:0/1694710252 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6eb36b620 con 0x55d6eb632800 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013f20 tx=0x7f54e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7f59cb1bc595dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e001fd30 con 0x55d6eb632800 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f54e002b480 con 0x55d6eb632800 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.788+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54d80316a0 con 0x55d6eb632800 2024-04-23T18:24:22.788 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54e002b680 con 0x55d6eb632800 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 10 client.16545 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d6eb641420 con 0x55d6eb632800 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d80316a0 con 0x55d6eb632800 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f54e0034860 con 0x55d6eb632800 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 10 client.16545 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 10 client.16545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 20 client.16545 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb642890 0x55d6eb644d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 4 client.16545 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6eb59a390 con 0x55d6eb642890 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.789+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb642890 0x55d6eb644d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.789 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.790+0000 7f54c7fff640 10 client.16545 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.790+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb642890 0x55d6eb644d00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3263227acb4d9f8a server_cookie=c12949637680363d in_seq=0 out_seq=0 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.790+0000 7f54c7fff640 1 -- 172.21.15.26:0/1694710252 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6eb59a390 con 0x55d6eb642890 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.790+0000 7f54c7fff640 10 client.16545 handle_command_reply: tid=0 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.790+0000 7f54f1ff2740 1 client.16545 shutdown 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 msgr2=0x55d6eb62fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013f20 tx=0x7f54e0004ab0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 shutdown_connections 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb642890 0x55d6eb644d00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb62fc70 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.790 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb62f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54c7fff640 0 client.16545 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1694710252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb632290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 >> 172.21.15.26:0/1694710252 conn(0x55d6eb623b30 msgr2=0x55d6eb629860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 shutdown_connections 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1694710252 wait complete. 2024-04-23T18:24:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.791+0000 7f54f1ff2740 20 client.16545 trim_cache size 0 max 16384 2024-04-23T18:24:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 Processor -- start 2024-04-23T18:24:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 -- start start 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb63e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb622b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81b0380 con 0x55d6eb632800 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81adb90 con 0x55d6eb627620 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81aba00 con 0x55d6eb627f40 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44150/0 (socket says 172.21.15.26:44150) 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54ed79b640 1 -- 172.21.15.26:0/1456052784 learned_addr learned my addr 172.21.15.26:0/1456052784 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.806+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb63e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54dffff640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb622b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 -- 172.21.15.26:0/1456052784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 msgr2=0x55d6eb622b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb622b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 -- 172.21.15.26:0/1456052784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 msgr2=0x55d6eb63e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb63e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 -- 172.21.15.26:0/1456052784 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6eb623070 con 0x55d6eb632800 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb63e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54dffff640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb622b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ed79b640 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013ef0 tx=0x7f54e0011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3324302e4d1e70b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ce7fc640 1 -- 172.21.15.26:0/1456052784 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e001fb30 con 0x55d6eb632800 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ce7fc640 1 -- 172.21.15.26:0/1456052784 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54e0034930 con 0x55d6eb632800 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.807+0000 7f54ce7fc640 1 -- 172.21.15.26:0/1456052784 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e002bb20 con 0x55d6eb632800 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1456052784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 msgr2=0x55d6eb64a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013ef0 tx=0x7f54e0011a80 comp rx=0 tx=0).stop 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1456052784 shutdown_connections 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb64a610 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627f40 0x55d6eb622b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/1456052784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627620 0x55d6eb63e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1456052784 >> 172.21.15.26:0/1456052784 conn(0x55d6eb623b30 msgr2=0x55d6eb622420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1456052784 shutdown_connections 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- 172.21.15.26:0/1456052784 wait complete. 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 -- start start 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.808+0000 7f54f1ff2740 1 Processor -- start 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627620 0x55d6eb638810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 0x55d6eb638d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ac050 con 0x55d6eb632800 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ad4c0 con 0x55d6eb627f40 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ac720 con 0x55d6eb627620 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 10 client.0 _ll_register_callbacks cb 0x7f54e0013dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ecf9a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627620 0x55d6eb638810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:44160/0 (socket says 172.21.15.26:44160) 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 -- 172.21.15.26:0/926084934 learned_addr learned my addr 172.21.15.26:0/926084934 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54dffff640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 0x55d6eb638d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 -- 172.21.15.26:0/926084934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627620 msgr2=0x55d6eb638810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627620 0x55d6eb638810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 -- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 msgr2=0x55d6eb638d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 0x55d6eb638d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 -- 172.21.15.26:0/926084934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6eb6a83d0 con 0x55d6eb632800 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54dffff640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 0x55d6eb638d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54ed79b640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f54e002b860 tx=0x7f54e001fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239dbfc104b9d89f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0043070 con 0x55d6eb632800 2024-04-23T18:24:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f54e0013460 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.809+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54e0034930 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54e0009d50 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 10 client.16551 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d6eb6a9550 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54e0034930 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f54e0005370 con 0x55d6eb632800 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 10 client.16551 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 10 client.16551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 20 client.16551 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb6a9b90 0x55d6eb6abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 4 client.16551 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6eb594200 con 0x55d6eb6a9b90 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.810+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb6a9b90 0x55d6eb6abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.811+0000 7f54c7fff640 10 client.16551 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.811+0000 7f54ecf9a640 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb6a9b90 0x55d6eb6abfb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee40a0a1f11bba8 server_cookie=c49d3e0ecc38b5db in_seq=0 out_seq=0 2024-04-23T18:24:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.837+0000 7f54c7fff640 1 -- 172.21.15.26:0/926084934 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6eb594200 con 0x55d6eb6a9b90 2024-04-23T18:24:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.837+0000 7f54c7fff640 10 client.16551 handle_command_reply: tid=0 2024-04-23T18:24:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.837+0000 7f54f1ff2740 1 client.16551 shutdown 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.837+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 msgr2=0x55d6eb6a7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.837+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f54e002b860 tx=0x7f54e001fd60 comp rx=0 tx=0).stop 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 shutdown_connections 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55d6eb6a9b90 0x55d6eb6abfb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d6eb627f40 0x55d6eb638d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55d6eb632800 0x55d6eb6a7e90 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 --2- 172.21.15.26:0/926084934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d6eb627620 0x55d6eb638810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54c7fff640 0 client.16551 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 >> 172.21.15.26:0/926084934 conn(0x55d6eb623b30 msgr2=0x55d6eb63d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 shutdown_connections 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 1 -- 172.21.15.26:0/926084934 wait complete. 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54f1ff2740 20 client.16551 trim_cache size 0 max 16384 2024-04-23T18:24:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:24:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 msgr2=0x7f54d00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 0x7f54d00a6280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54e0013770 tx=0x7f54e0013b70 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 msgr2=0x7f54e8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c920 tx=0x7f54e400cde0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 shutdown_connections 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54d00a3dc0 0x7f54d00a6280 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54e8107290 0x7f54e8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54e819cf30 0x7f54e819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 --2- 172.21.15.26:0/667375269 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54e81021c0 0x7f54e8198670 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.838+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 >> 172.21.15.26:0/667375269 conn(0x7f54e80fda20 msgr2=0x7f54e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.839+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 shutdown_connections 2024-04-23T18:24:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:22.839+0000 7f54ef1fe640 1 -- 172.21.15.26:0/667375269 wait complete. 2024-04-23T18:24:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:23 smithi110 ceph-mon[32585]: pgmap v222: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:23.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:23 smithi026 ceph-mon[29364]: pgmap v222: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:23.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:23 smithi131 ceph-mon[33041]: pgmap v222: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:23.680 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.93 MB/sec execute 258 sec latency 2878.800 ms 2024-04-23T18:24:24.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.91 MB/sec execute 259 sec latency 3878.841 ms 2024-04-23T18:24:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:25 smithi110 ceph-mon[32585]: pgmap v223: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:25 smithi026 ceph-mon[29364]: pgmap v223: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:25 smithi131 ceph-mon[33041]: pgmap v223: 97 pgs: 97 active+clean; 261 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 284 op/s 2024-04-23T18:24:25.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.88 MB/sec execute 260 sec latency 4878.908 ms 2024-04-23T18:24:26.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.86 MB/sec execute 261 sec latency 5879.019 ms 2024-04-23T18:24:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:27 smithi110 ceph-mon[32585]: pgmap v224: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-04-23T18:24:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:27 smithi026 ceph-mon[29364]: pgmap v224: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-04-23T18:24:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:27 smithi131 ceph-mon[33041]: pgmap v224: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 186 op/s 2024-04-23T18:24:27.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.84 MB/sec execute 262 sec latency 6879.162 ms 2024-04-23T18:24:28.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.82 MB/sec execute 263 sec latency 7879.255 ms 2024-04-23T18:24:29.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:29 smithi110 ceph-mon[32585]: pgmap v225: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-23T18:24:29.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:29 smithi026 ceph-mon[29364]: pgmap v225: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-23T18:24:29.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:29 smithi131 ceph-mon[33041]: pgmap v225: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-23T18:24:29.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.79 MB/sec execute 264 sec latency 8879.332 ms 2024-04-23T18:24:30.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.77 MB/sec execute 265 sec latency 9879.423 ms 2024-04-23T18:24:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:31 smithi110 ceph-mon[32585]: pgmap v226: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-23T18:24:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:31 smithi026 ceph-mon[29364]: pgmap v226: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-23T18:24:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:31 smithi131 ceph-mon[33041]: pgmap v226: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-23T18:24:31.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.75 MB/sec execute 266 sec latency 10879.546 ms 2024-04-23T18:24:32.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.73 MB/sec execute 267 sec latency 11879.627 ms 2024-04-23T18:24:33.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:33 smithi110 ceph-mon[32585]: pgmap v227: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-23T18:24:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:33 smithi026 ceph-mon[29364]: pgmap v227: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-23T18:24:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:33 smithi131 ceph-mon[33041]: pgmap v227: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-23T18:24:33.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.71 MB/sec execute 268 sec latency 12879.741 ms 2024-04-23T18:24:34.681 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.69 MB/sec execute 269 sec latency 13879.803 ms 2024-04-23T18:24:35.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:35 smithi110 ceph-mon[32585]: pgmap v228: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 126 KiB/s wr, 45 op/s 2024-04-23T18:24:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:35 smithi026 ceph-mon[29364]: pgmap v228: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 126 KiB/s wr, 45 op/s 2024-04-23T18:24:35.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:35 smithi131 ceph-mon[33041]: pgmap v228: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 126 KiB/s wr, 45 op/s 2024-04-23T18:24:35.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.67 MB/sec execute 270 sec latency 14879.873 ms 2024-04-23T18:24:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 -- 172.21.15.26:0/3440960178 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 msgr2=0x7f01841014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 --2- 172.21.15.26:0/3440960178 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f01841014d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0174009900 tx=0x7f017402a410 comp rx=0 tx=0).stop 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 -- 172.21.15.26:0/3440960178 shutdown_connections 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 --2- 172.21.15.26:0/3440960178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 0x7f018410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 --2- 172.21.15.26:0/3440960178 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0184101aa0 0x7f0184105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 --2- 172.21.15.26:0/3440960178 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f01841014d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 -- 172.21.15.26:0/3440960178 >> 172.21.15.26:0/3440960178 conn(0x7f0184077fd0 msgr2=0x7f01840783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 -- 172.21.15.26:0/3440960178 shutdown_connections 2024-04-23T18:24:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.932+0000 7f0188d20640 1 -- 172.21.15.26:0/3440960178 wait complete. 2024-04-23T18:24:35.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 Processor -- start 2024-04-23T18:24:35.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 -- start start 2024-04-23T18:24:35.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 0x7f018410ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.929 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f018410f150 0x7f01841a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0184112230 con 0x7f01841010f0 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01841120b0 con 0x7f0184106170 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f0188d20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f01841123b0 con 0x7f018410f150 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.933+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57016/0 (socket says 172.21.15.26:57016) 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 -- 172.21.15.26:0/1436090248 learned_addr learned my addr 172.21.15.26:0/1436090248 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f017bfff640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 0x7f018410ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f0182d9d640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f018410f150 0x7f01841a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 -- 172.21.15.26:0/1436090248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f018410f150 msgr2=0x7f01841a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f018410f150 0x7f01841a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 -- 172.21.15.26:0/1436090248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 msgr2=0x7f018410ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 0x7f018410ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 -- 172.21.15.26:0/1436090248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01841a6f60 con 0x7f01841010f0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f018259c640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f01740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f017b7fe640 1 -- 172.21.15.26:0/1436090248 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017403f030 con 0x7f01841010f0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01841a71f0 con 0x7f01841010f0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f017b7fe640 1 -- 172.21.15.26:0/1436090248 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0174003800 con 0x7f01841010f0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f017b7fe640 1 -- 172.21.15.26:0/1436090248 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174035070 con 0x7f01841010f0 2024-04-23T18:24:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.934+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01841a76e0 con 0x7f01841010f0 2024-04-23T18:24:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.936+0000 7f017b7fe640 1 -- 172.21.15.26:0/1436090248 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0174003cb0 con 0x7f01841010f0 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 Processor -- start 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 -- start start 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 0x55e220e0aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e17bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e220d6f970 con 0x55e220e0af40 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e220d0c4a0 con 0x55e220e0a620 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018ba36740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e220d6b930 con 0x55e220e15800 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f017b7fe640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 0x7f01700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57018/0 (socket says 172.21.15.26:57018) 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f018259c640 1 -- 172.21.15.26:0/4079901303 learned_addr learned my addr 172.21.15.26:0/4079901303 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.937+0000 7f0182d9d640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 0x55e220e0aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0182d9d640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 0x7f01700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f017bfff640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e17bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f017b7fe640 1 -- 172.21.15.26:0/1436090248 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f01740f1140 con 0x7f01841010f0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 -- 172.21.15.26:0/4079901303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 msgr2=0x55e220e17bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e17bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 -- 172.21.15.26:0/4079901303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 msgr2=0x55e220e0aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 0x55e220e0aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 -- 172.21.15.26:0/4079901303 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e220a82210 con 0x55e220e0af40 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0182d9d640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 0x55e220e0aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f018259c640 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f01740bd2a0 tx=0x7f017402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594bb9c48ff4a27e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0182d9d640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 0x7f01700a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f017c013d20 tx=0x7f017c004a40 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0178ff9640 1 -- 172.21.15.26:0/4079901303 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049070 con 0x55e220e0af40 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0178ff9640 1 -- 172.21.15.26:0/4079901303 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01740bd7e0 con 0x55e220e0af40 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.938+0000 7f0178ff9640 1 -- 172.21.15.26:0/4079901303 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01740eeb10 con 0x55e220e0af40 2024-04-23T18:24:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- 172.21.15.26:0/4079901303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 msgr2=0x55e220e152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f01740bd2a0 tx=0x7f017402a410 comp rx=0 tx=0).stop 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- 172.21.15.26:0/4079901303 shutdown_connections 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e17bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0af40 0x55e220e152c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 --2- 172.21.15.26:0/4079901303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0a620 0x55e220e0aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- 172.21.15.26:0/4079901303 >> 172.21.15.26:0/4079901303 conn(0x55e220e06b30 msgr2=0x55e220e06f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- 172.21.15.26:0/4079901303 shutdown_connections 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- 172.21.15.26:0/4079901303 wait complete. 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.939+0000 7f018ba36740 1 -- start start 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 Processor -- start 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 0x55e220e123b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55e220e181f0 con 0x55e220e0a620 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e220d0a0b0 con 0x55e220e0af40 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55e220e18370 con 0x55e220e15800 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018ba36740 10 client.0 _ll_register_callbacks cb 0x55e220cf7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57020/0 (socket says 172.21.15.26:57020) 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 -- 172.21.15.26:0/522205869 learned_addr learned my addr 172.21.15.26:0/522205869 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f0182d9d640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f017bfff640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 0x55e220e123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 -- 172.21.15.26:0/522205869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 msgr2=0x55e220e128f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e128f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 -- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 msgr2=0x55e220e123b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 0x55e220e123b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 -- 172.21.15.26:0/522205869 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e220b4e620 con 0x55e220e0a620 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f017bfff640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 0x55e220e123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f018259c640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f01740ee940 tx=0x7f01740095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd976b29e9c552d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f0169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.940+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049040 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f01740eeb10 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01740bd7e0 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01740c1490 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 10 client.16563 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e220e181f0 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01740bd7e0 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f01740c58c0 con 0x55e220e0a620 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 10 client.16563 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 20 client.16563 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e25330 0x55e220e277a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 4 client.16563 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e220d7d390 con 0x55e220e25330 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.941+0000 7f017bfff640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e25330 0x55e220e277a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.942+0000 7f0169ffb640 10 client.16563 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.942+0000 7f017bfff640 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e25330 0x55e220e277a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba28043549bb641b server_cookie=d37f495fc55abf2f in_seq=0 out_seq=0 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f0169ffb640 1 -- 172.21.15.26:0/522205869 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e220d7d390 con 0x55e220e25330 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f0169ffb640 10 client.16563 handle_command_reply: tid=0 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 client.16563 shutdown 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 msgr2=0x55e220e150c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f01740ee940 tx=0x7f01740095e0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 shutdown_connections 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e25330 0x55e220e277a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e0af40 0x55e220e123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e15800 0x55e220e128f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 --2- 172.21.15.26:0/522205869 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e150c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f0169ffb640 0 client.16563 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 >> 172.21.15.26:0/522205869 conn(0x55e220e06b30 msgr2=0x55e220e0c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 shutdown_connections 2024-04-23T18:24:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 1 -- 172.21.15.26:0/522205869 wait complete. 2024-04-23T18:24:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.943+0000 7f018ba36740 20 client.16563 trim_cache size 0 max 16384 2024-04-23T18:24:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 Processor -- start 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 -- start start 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e2d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e2d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f01841b2690 con 0x55e220e0a620 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01841afea0 con 0x55e220e15800 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018ba36740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f01841add10 con 0x55e220e0af40 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57030/0 (socket says 172.21.15.26:57030) 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.958+0000 7f018259c640 1 -- 172.21.15.26:0/1565107165 learned_addr learned my addr 172.21.15.26:0/1565107165 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f0182d9d640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e2d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f017bfff640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e2d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 -- 172.21.15.26:0/1565107165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 msgr2=0x55e220e2d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e2d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 -- 172.21.15.26:0/1565107165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 msgr2=0x55e220e2d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e2d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 -- 172.21.15.26:0/1565107165 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e220e12320 con 0x55e220e0a620 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f0182d9d640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e2d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018259c640 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f01740bb5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999b1fd048214ce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f016bfff640 1 -- 172.21.15.26:0/1565107165 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049070 con 0x55e220e0a620 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f016bfff640 1 -- 172.21.15.26:0/1565107165 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01740bd7e0 con 0x55e220e0a620 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f016bfff640 1 -- 172.21.15.26:0/1565107165 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01740c10d0 con 0x55e220e0a620 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018ba36740 1 -- 172.21.15.26:0/1565107165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 msgr2=0x55e220e20fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.959+0000 7f018ba36740 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f01740bb5f0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- 172.21.15.26:0/1565107165 shutdown_connections 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e2d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e2d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- 172.21.15.26:0/1565107165 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e20fb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- 172.21.15.26:0/1565107165 >> 172.21.15.26:0/1565107165 conn(0x55e220e06b30 msgr2=0x55e220e05420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- 172.21.15.26:0/1565107165 shutdown_connections 2024-04-23T18:24:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- 172.21.15.26:0/1565107165 wait complete. 2024-04-23T18:24:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- start start 2024-04-23T18:24:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 Processor -- start 2024-04-23T18:24:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e1a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e1ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f01841ae360 con 0x55e220e0a620 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01841af7d0 con 0x55e220e15800 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f01841aea30 con 0x55e220e0af40 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018ba36740 10 client.0 _ll_register_callbacks cb 0x7f0174005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57038/0 (socket says 172.21.15.26:57038) 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f018259c640 1 -- 172.21.15.26:0/4083554889 learned_addr learned my addr 172.21.15.26:0/4083554889 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.960+0000 7f0182d9d640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e1ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 -- 172.21.15.26:0/4083554889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 msgr2=0x55e220e1a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f017bfff640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e1a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e1a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 -- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 msgr2=0x55e220e1ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e1ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 -- 172.21.15.26:0/4083554889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e220e89b20 con 0x55e220e0a620 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f017bfff640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e1a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018259c640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f01740bcde0 tx=0x7f0174009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae58e62de3a8e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0174049040 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f01740c58c0 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01740bd7e0 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01740c5d50 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 10 client.16569 fetch_fsmap learned FSMap version 37 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e220e89b20 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01740bd7e0 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f01740b8e30 con 0x55e220e0a620 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 10 client.16569 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 20 client.16569 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.961+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e8b0c0 0x55e220e8d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.962+0000 7f018ba36740 4 client.16569 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.962+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e220d77200 con 0x55e220e8b0c0 2024-04-23T18:24:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.962+0000 7f017bfff640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e8b0c0 0x55e220e8d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.962+0000 7f017bfff640 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e8b0c0 0x55e220e8d4b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1085da8045813d3f server_cookie=751d3d35d0c813fc in_seq=0 out_seq=0 2024-04-23T18:24:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.962+0000 7f0169ffb640 10 client.16569 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f0169ffb640 1 -- 172.21.15.26:0/4083554889 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e220d77200 con 0x55e220e8b0c0 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f0169ffb640 10 client.16569 handle_command_reply: tid=0 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 client.16569 shutdown 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 msgr2=0x55e220e1a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f01740bcde0 tx=0x7f0174009650 comp rx=0 tx=0).stop 2024-04-23T18:24:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 shutdown_connections 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55e220e8b0c0 0x55e220e8d4b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e220e15800 0x55e220e1ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e220e0af40 0x55e220e1a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f0169ffb640 0 client.16569 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 --2- 172.21.15.26:0/4083554889 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55e220e0a620 0x55e220e1a0d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 >> 172.21.15.26:0/4083554889 conn(0x55e220e06b30 msgr2=0x55e220e05cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 shutdown_connections 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 1 -- 172.21.15.26:0/4083554889 wait complete. 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.976+0000 7f018ba36740 20 client.16569 trim_cache size 0 max 16384 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:24:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 msgr2=0x7f01700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 0x7f01700a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f017c013d20 tx=0x7f017c004a40 comp rx=0 tx=0).stop 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 msgr2=0x7f018410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f01740042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 shutdown_connections 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f01700a3dc0 0x7f01700a6280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0184106170 0x7f018410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f018410f150 0x7f01841a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 --2- 172.21.15.26:0/1436090248 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f01841010f0 0x7f018410e6d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 >> 172.21.15.26:0/1436090248 conn(0x7f0184077fd0 msgr2=0x7f01840fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 shutdown_connections 2024-04-23T18:24:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:35.977+0000 7f0188d20640 1 -- 172.21.15.26:0/1436090248 wait complete. 2024-04-23T18:24:36.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.64 MB/sec execute 271 sec latency 15879.987 ms 2024-04-23T18:24:37.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:37 smithi110 ceph-mon[32585]: pgmap v229: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 211 KiB/s wr, 49 op/s 2024-04-23T18:24:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:37 smithi026 ceph-mon[29364]: pgmap v229: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 211 KiB/s wr, 49 op/s 2024-04-23T18:24:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:37 smithi131 ceph-mon[33041]: pgmap v229: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 211 KiB/s wr, 49 op/s 2024-04-23T18:24:37.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.62 MB/sec execute 272 sec latency 16880.037 ms 2024-04-23T18:24:38.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.60 MB/sec execute 273 sec latency 17880.115 ms 2024-04-23T18:24:39.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:39 smithi110 ceph-mon[32585]: pgmap v230: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:39 smithi026 ceph-mon[29364]: pgmap v230: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:39.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:39 smithi131 ceph-mon[33041]: pgmap v230: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:39.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.58 MB/sec execute 274 sec latency 18880.173 ms 2024-04-23T18:24:40.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.56 MB/sec execute 275 sec latency 19880.323 ms 2024-04-23T18:24:41.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:41 smithi110 ceph-mon[32585]: pgmap v231: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:41.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:41 smithi026 ceph-mon[29364]: pgmap v231: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:41.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:41 smithi131 ceph-mon[33041]: pgmap v231: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 103 KiB/s wr, 11 op/s 2024-04-23T18:24:41.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.54 MB/sec execute 276 sec latency 20880.439 ms 2024-04-23T18:24:42.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.52 MB/sec execute 277 sec latency 21880.542 ms 2024-04-23T18:24:43.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:43 smithi110 ceph-mon[32585]: pgmap v232: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 617 KiB/s wr, 36 op/s 2024-04-23T18:24:43.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:43 smithi026 ceph-mon[29364]: pgmap v232: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 617 KiB/s wr, 36 op/s 2024-04-23T18:24:43.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:43 smithi131 ceph-mon[33041]: pgmap v232: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 617 KiB/s wr, 36 op/s 2024-04-23T18:24:43.682 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.50 MB/sec execute 278 sec latency 22880.690 ms 2024-04-23T18:24:44.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:44 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:44.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:44 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:44.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:44 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:44.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.48 MB/sec execute 279 sec latency 23880.753 ms 2024-04-23T18:24:45.392 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:45 smithi131 ceph-mon[33041]: pgmap v233: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-04-23T18:24:45.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:45 smithi110 ceph-mon[32585]: pgmap v233: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-04-23T18:24:45.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:45 smithi026 ceph-mon[29364]: pgmap v233: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 600 KiB/s wr, 29 op/s 2024-04-23T18:24:45.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.46 MB/sec execute 280 sec latency 24880.888 ms 2024-04-23T18:24:46.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.44 MB/sec execute 281 sec latency 25880.975 ms 2024-04-23T18:24:47.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:47 smithi110 ceph-mon[32585]: pgmap v234: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:24:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:47 smithi026 ceph-mon[29364]: pgmap v234: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:24:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:47 smithi131 ceph-mon[33041]: pgmap v234: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:24:47.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.42 MB/sec execute 282 sec latency 26881.054 ms 2024-04-23T18:24:48.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.41 MB/sec execute 283 sec latency 27881.136 ms 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.077+0000 7fb8a0336640 1 -- 172.21.15.26:0/1090959512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8981063a0 msgr2=0x7fb89810df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.077+0000 7fb8a0336640 1 --2- 172.21.15.26:0/1090959512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8981063a0 0x7fb89810df70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb890009ca0 tx=0x7fb89002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 -- 172.21.15.26:0/1090959512 shutdown_connections 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 --2- 172.21.15.26:0/1090959512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981021a0 0x7fb898105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 --2- 172.21.15.26:0/1090959512 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb8981063a0 0x7fb89810df70 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 --2- 172.21.15.26:0/1090959512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981017f0 0x7fb898101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 -- 172.21.15.26:0/1090959512 >> 172.21.15.26:0/1090959512 conn(0x7fb898077390 msgr2=0x7fb8980fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 -- 172.21.15.26:0/1090959512 shutdown_connections 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 -- 172.21.15.26:0/1090959512 wait complete. 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 Processor -- start 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.078+0000 7fb8a0336640 1 -- start start 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981017f0 0x7fb89819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981063a0 0x7fb89819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981112f0 con 0x7fb898198670 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb898111170 con 0x7fb8981017f0 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb8a0336640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb898111470 con 0x7fb8981063a0 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37704/0 (socket says 172.21.15.26:37704) 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 -- 172.21.15.26:0/499110389 learned_addr learned my addr 172.21.15.26:0/499110389 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981063a0 0x7fb89819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981017f0 0x7fb89819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 -- 172.21.15.26:0/499110389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981063a0 msgr2=0x7fb89819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981063a0 0x7fb89819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 -- 172.21.15.26:0/499110389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981017f0 msgr2=0x7fb89819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981017f0 0x7fb89819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 -- 172.21.15.26:0/499110389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb898199270 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb89e8d3640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb890005cc0 tx=0x7fb89002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.079+0000 7fb88f7fe640 1 -- 172.21.15.26:0/499110389 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890003bf0 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.080+0000 7fb88f7fe640 1 -- 172.21.15.26:0/499110389 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb890003d90 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.080+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb898199500 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.080+0000 7fb88f7fe640 1 -- 172.21.15.26:0/499110389 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89002fb20 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.080+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8981a56a0 con 0x7fb898198670 2024-04-23T18:24:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.082+0000 7fb88f7fe640 1 -- 172.21.15.26:0/499110389 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb890004580 con 0x7fb898198670 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.082+0000 7fb8a3045740 1 Processor -- start 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 -- start start 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b661923a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661930bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b661888970 con 0x55b661923f40 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b6618254a0 con 0x55b66192e800 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb8a3045740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b661884930 con 0x55b661923620 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb88f7fe640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89e0d2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89e0d2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37718/0 (socket says 172.21.15.26:37718) 2024-04-23T18:24:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89e8d3640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3621017816 learned_addr learned my addr 172.21.15.26:0/3621017816 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b661923a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661930bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.083+0000 7fb88f7fe640 1 -- 172.21.15.26:0/499110389 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7fb8900f1c00 con 0x7fb898198670 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3621017816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 msgr2=0x55b661923a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b661923a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3621017816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 msgr2=0x55b661930bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661930bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3621017816 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66159b210 con 0x55b661923f40 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661930bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e8d3640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 0x7fb8840a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8900bdbe0 tx=0x7fb89008f040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb89400ad00 tx=0x7fb89400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486f15664810b261 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb88cbfe640 1 -- 172.21.15.26:0/3621017816 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89401d070 con 0x55b661923f40 2024-04-23T18:24:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb88cbfe640 1 -- 172.21.15.26:0/3621017816 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb894018af0 con 0x55b661923f40 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.084+0000 7fb88cbfe640 1 -- 172.21.15.26:0/3621017816 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894010b30 con 0x55b661923f40 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- 172.21.15.26:0/3621017816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 msgr2=0x55b66192e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb89400ad00 tx=0x7fb89400f790 comp rx=0 tx=0).stop 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- 172.21.15.26:0/3621017816 shutdown_connections 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661930bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b66192e2c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3621017816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b661923a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- 172.21.15.26:0/3621017816 >> 172.21.15.26:0/3621017816 conn(0x55b66191fb30 msgr2=0x55b66191ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- 172.21.15.26:0/3621017816 shutdown_connections 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- 172.21.15.26:0/3621017816 wait complete. 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 -- start start 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.085+0000 7fb8a3045740 1 Processor -- start 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923f40 0x55b66192b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b66192e800 0x55b66192ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b661931430 con 0x55b661923620 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b6618230b0 con 0x55b661923f40 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b6619315b0 con 0x55b66192e800 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 10 client.0 _ll_register_callbacks cb 0x55b661810d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37728/0 (socket says 172.21.15.26:37728) 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 -- 172.21.15.26:0/474544500 learned_addr learned my addr 172.21.15.26:0/474544500 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923f40 0x55b66192b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b66192e800 0x55b66192ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 -- 172.21.15.26:0/474544500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b66192e800 msgr2=0x55b66192ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b66192e800 0x55b66192ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 -- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923f40 msgr2=0x55b66192b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923f40 0x55b66192b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 -- 172.21.15.26:0/474544500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b661667620 con 0x55b661923620 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb8880042a0 tx=0x7fb8880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3498dc5d6e57f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888005220 con 0x55b661923620 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7fb8880053c0 con 0x55b661923620 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.086+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb894018af0 con 0x55b661923620 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb8880055c0 con 0x55b661923620 2024-04-23T18:24:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 10 client.16581 fetch_fsmap learned FSMap version 38 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b66193d380 con 0x55b661923620 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb894018af0 con 0x55b661923620 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 38) ==== 8358+0+0 (secure 0 0 0) 0x7fb888012920 con 0x55b661923620 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 10 client.16581 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 20 client.16581 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b66193e7f0 0x55b661940c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 4 client.16581 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b661896390 con 0x55b66193e7f0 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.087+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b66193e7f0 0x55b661940c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.088+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b66193e7f0 0x55b661940c40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e45d6a37d4219 server_cookie=b6cc56bd6776d3f9 in_seq=0 out_seq=0 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.088+0000 7fb881ffb640 10 client.16581 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb881ffb640 1 -- 172.21.15.26:0/474544500 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b661896390 con 0x55b66193e7f0 2024-04-23T18:24:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb881ffb640 10 client.16581 handle_command_reply: tid=0 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 client.16581 shutdown 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 msgr2=0x55b66192e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb8880042a0 tx=0x7fb8880046c0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 shutdown_connections 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b66193e7f0 0x55b661940c40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923f40 0x55b66192b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b66192e800 0x55b66192ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb881ffb640 0 client.16581 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 --2- 172.21.15.26:0/474544500 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923620 0x55b66192e190 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 >> 172.21.15.26:0/474544500 conn(0x55b66191fb30 msgr2=0x55b661925860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 shutdown_connections 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 1 -- 172.21.15.26:0/474544500 wait complete. 2024-04-23T18:24:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.089+0000 7fb8a3045740 20 client.16581 trim_cache size 0 max 16384 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 Processor -- start 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 -- start start 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923620 0x55b66193a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923f40 0x55b661946550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981b0430 con 0x55b66192e800 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981adc40 con 0x55b661923620 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.104+0000 7fb8a3045740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981abab0 con 0x55b661923f40 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37734/0 (socket says 172.21.15.26:37734) 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 -- 172.21.15.26:0/563344501 learned_addr learned my addr 172.21.15.26:0/563344501 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923f40 0x55b661946550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923620 0x55b66193a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 -- 172.21.15.26:0/563344501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923f40 msgr2=0x55b661946550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923f40 0x55b661946550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 -- 172.21.15.26:0/563344501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923620 msgr2=0x55b66193a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923620 0x55b66193a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 -- 172.21.15.26:0/563344501 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66192a450 con 0x55b66192e800 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b6b0 tx=0x7fb888005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebb618f8517a746 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb883fff640 1 -- 172.21.15.26:0/563344501 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888012840 con 0x55b66192e800 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb883fff640 1 -- 172.21.15.26:0/563344501 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8880129e0 con 0x55b66192e800 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.105+0000 7fb883fff640 1 -- 172.21.15.26:0/563344501 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888010690 con 0x55b66192e800 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- 172.21.15.26:0/563344501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 msgr2=0x55b661946a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b6b0 tx=0x7fb888005760 comp rx=0 tx=0).stop 2024-04-23T18:24:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- 172.21.15.26:0/563344501 shutdown_connections 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b66192e800 0x55b661946a90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923f40 0x55b661946550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 --2- 172.21.15.26:0/563344501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b661923620 0x55b66193a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- 172.21.15.26:0/563344501 >> 172.21.15.26:0/563344501 conn(0x55b66191fb30 msgr2=0x55b66191e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- 172.21.15.26:0/563344501 shutdown_connections 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- 172.21.15.26:0/563344501 wait complete. 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 -- start start 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.106+0000 7fb8a3045740 1 Processor -- start 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b6619368f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661934330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ac100 con 0x55b661923f40 2024-04-23T18:24:49.104 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ad570 con 0x55b66192e800 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ac7d0 con 0x55b661923620 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 10 client.0 _ll_register_callbacks cb 0x7fb88800bd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37742/0 (socket says 172.21.15.26:37742) 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3021248809 learned_addr learned my addr 172.21.15.26:0/3021248809 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b6619368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661934330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3021248809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 msgr2=0x55b6619368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b6619368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 msgr2=0x55b661934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661934330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 -- 172.21.15.26:0/3021248809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b661930f60 con 0x55b661923f40 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661934330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb89e0d2640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb89400ad00 tx=0x7fb894005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e6104fd6db1a59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89401d040 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7fb8940187c0 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.107+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8880129e0 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb8940189c0 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 10 client.16587 fetch_fsmap learned FSMap version 38 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b6619a4960 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8880129e0 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 38) ==== 8358+0+0 (secure 0 0 0) 0x7fb894010a30 con 0x55b661923f40 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 10 client.16587 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 20 client.16587 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b6619a4e00 0x55b6619a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 4 client.16587 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b661890200 con 0x55b6619a4e00 2024-04-23T18:24:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.108+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b6619a4e00 0x55b6619a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.109+0000 7fb881ffb640 10 client.16587 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.109+0000 7fb89d8d1640 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b6619a4e00 0x55b6619a7220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e070adaa89742b24 server_cookie=5bf19c1136b57678 in_seq=0 out_seq=0 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb881ffb640 1 -- 172.21.15.26:0/3021248809 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b661890200 con 0x55b6619a4e00 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb881ffb640 10 client.16587 handle_command_reply: tid=0 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb8a3045740 1 client.16587 shutdown 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 msgr2=0x55b661933df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb89400ad00 tx=0x7fb894005e60 comp rx=0 tx=0).stop 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 shutdown_connections 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.125+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b6619a4e00 0x55b6619a7220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b66192e800 0x55b661934330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b661923f40 0x55b661933df0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb881ffb640 0 client.16587 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 --2- 172.21.15.26:0/3021248809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b661923620 0x55b6619368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 >> 172.21.15.26:0/3021248809 conn(0x55b66191fb30 msgr2=0x55b661939b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 shutdown_connections 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 1 -- 172.21.15.26:0/3021248809 wait complete. 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a3045740 20 client.16587 trim_cache size 0 max 16384 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:24:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:24:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 msgr2=0x7fb8840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 0x7fb8840a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8900bdbe0 tx=0x7fb89008f040 comp rx=0 tx=0).stop 2024-04-23T18:24:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 msgr2=0x7fb898198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb890005cc0 tx=0x7fb89002f380 comp rx=0 tx=0).stop 2024-04-23T18:24:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 shutdown_connections 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb898198670 0x7fb898198af0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8981063a0 0x7fb89819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 --2- 172.21.15.26:0/499110389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8981017f0 0x7fb89819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 >> 172.21.15.26:0/499110389 conn(0x7fb898077390 msgr2=0x7fb8981007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.126+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 shutdown_connections 2024-04-23T18:24:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:24:49.127+0000 7fb8a0336640 1 -- 172.21.15.26:0/499110389 wait complete. 2024-04-23T18:24:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:49 smithi110 ceph-mon[32585]: pgmap v235: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:49.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:49 smithi026 ceph-mon[29364]: pgmap v235: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:49.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:49 smithi131 ceph-mon[33041]: pgmap v235: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:49.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.39 MB/sec execute 284 sec latency 28881.210 ms 2024-04-23T18:24:50.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.37 MB/sec execute 285 sec latency 29881.322 ms 2024-04-23T18:24:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:51 smithi110 ceph-mon[32585]: pgmap v236: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:51 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:51 smithi026 ceph-mon[29364]: pgmap v236: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:51.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:51 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:51 smithi131 ceph-mon[33041]: pgmap v236: 97 pgs: 97 active+clean; 254 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.0 MiB/s wr, 48 op/s 2024-04-23T18:24:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:51 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.35 MB/sec execute 286 sec latency 30881.383 ms 2024-04-23T18:24:52.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.33 MB/sec execute 287 sec latency 31881.501 ms 2024-04-23T18:24:53.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:53 smithi110 ceph-mon[32585]: pgmap v237: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-04-23T18:24:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:53 smithi026 ceph-mon[29364]: pgmap v237: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-04-23T18:24:53.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:53 smithi131 ceph-mon[33041]: pgmap v237: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-04-23T18:24:53.683 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.31 MB/sec execute 288 sec latency 32881.637 ms 2024-04-23T18:24:54.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.29 MB/sec execute 289 sec latency 33881.747 ms 2024-04-23T18:24:55.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:55 smithi110 ceph-mon[32585]: pgmap v238: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.2 MiB/s wr, 53 op/s 2024-04-23T18:24:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:55 smithi026 ceph-mon[29364]: pgmap v238: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.2 MiB/s wr, 53 op/s 2024-04-23T18:24:55.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:55 smithi131 ceph-mon[33041]: pgmap v238: 97 pgs: 97 active+clean; 253 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.2 MiB/s wr, 53 op/s 2024-04-23T18:24:55.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440312 5.27 MB/sec execute 290 sec latency 34881.879 ms 2024-04-23T18:24:55.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 -- 172.21.15.26:0/2592080790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 msgr2=0x7f75f0103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 --2- 172.21.15.26:0/2592080790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0103610 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009900 tx=0x7f75e002a480 comp rx=0 tx=0).stop 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 -- 172.21.15.26:0/2592080790 shutdown_connections 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 --2- 172.21.15.26:0/2592080790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f010c570 0x7f75f010e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 --2- 172.21.15.26:0/2592080790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0103b50 0x7f75f010c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 --2- 172.21.15.26:0/2592080790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0103610 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 -- 172.21.15.26:0/2592080790 >> 172.21.15.26:0/2592080790 conn(0x7f75f0077fb0 msgr2=0x7f75f00783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 -- 172.21.15.26:0/2592080790 shutdown_connections 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.992+0000 7f75f60a0640 1 -- 172.21.15.26:0/2592080790 wait complete. 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 Processor -- start 2024-04-23T18:24:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 -- start start 2024-04-23T18:24:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 0x7f75f01013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 0x7f75f0101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f75f01121b0 con 0x7f75f0103230 2024-04-23T18:24:55.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0112030 con 0x7f75f0103b50 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75f60a0640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0112330 con 0x7f75f010c570 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75ef7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75ef7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37762/0 (socket says 172.21.15.26:37762) 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75ef7fe640 1 -- 172.21.15.26:0/465338867 learned_addr learned my addr 172.21.15.26:0/465338867 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.993+0000 7f75eeffd640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 0x7f75f01013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75effff640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 0x7f75f0101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 -- 172.21.15.26:0/465338867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 msgr2=0x7f75f0101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 0x7f75f0101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 -- 172.21.15.26:0/465338867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 msgr2=0x7f75f01013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 0x7f75f01013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 -- 172.21.15.26:0/465338867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f01a6c30 con 0x7f75f0103230 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75effff640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 0x7f75f0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75eeffd640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 0x7f75f01013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ef7fe640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f75e0006010 tx=0x7f75e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:55.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e003f030 con 0x7f75f0103230 2024-04-23T18:24:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f01a6ec0 con 0x7f75f0103230 2024-04-23T18:24:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e0003800 con 0x7f75f0103230 2024-04-23T18:24:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0035070 con 0x7f75f0103230 2024-04-23T18:24:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.994+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f01a7320 con 0x7f75f0103230 2024-04-23T18:24:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.995+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f00691c0 con 0x7f75f0103230 2024-04-23T18:24:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.996+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f75e0003cb0 con 0x7f75f0103230 2024-04-23T18:24:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.997+0000 7f75ecff9640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 0x7f75d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.998+0000 7f75eeffd640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 0x7f75d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.998+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f75e00f1210 con 0x7f75f0103230 2024-04-23T18:24:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:55.998+0000 7f75eeffd640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 0x7f75d80a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75f0102910 tx=0x7f75e4005ec0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.002+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75e00bd8b0 con 0x7f75f0103230 2024-04-23T18:24:56.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.234+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f0107160 con 0x7f75f0103230 2024-04-23T18:24:56.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.235+0000 7f75ecff9640 1 -- 172.21.15.26:0/465338867 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11889 (secure 0 0 0) 0x7f75e00bd1b0 con 0x7f75f0103230 2024-04-23T18:24:56.228 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:24:56.228 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:24:49.420220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:24:56.228 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 39 2024-04-23T18:24:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 msgr2=0x7f75d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 0x7f75d80a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75f0102910 tx=0x7f75e4005ec0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 msgr2=0x7f75f0100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f75e0006010 tx=0x7f75e00042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 shutdown_connections 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f75d80a3d20 0x7f75d80a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f0103b50 0x7f75f01013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f010c570 0x7f75f0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 --2- 172.21.15.26:0/465338867 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f75f0103230 0x7f75f0100e80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 >> 172.21.15.26:0/465338867 conn(0x7f75f0077fb0 msgr2=0x7f75f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 shutdown_connections 2024-04-23T18:24:56.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.237+0000 7f75f60a0640 1 -- 172.21.15.26:0/465338867 wait complete. 2024-04-23T18:24:56.241 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 1e98f5fd --release --await' 2024-04-23T18:24:56.242 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 1e98f5fd --release --await 2024-04-23T18:24:56.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 -- 172.21.15.26:0/3980271557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 msgr2=0x7f72d0101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 --2- 172.21.15.26:0/3980271557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0101480 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009900 tx=0x7f72c002a410 comp rx=0 tx=0).stop 2024-04-23T18:24:56.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 -- 172.21.15.26:0/3980271557 shutdown_connections 2024-04-23T18:24:56.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 --2- 172.21.15.26:0/3980271557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d0106170 0x7f72d010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 --2- 172.21.15.26:0/3980271557 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0101a50 0x7f72d0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 --2- 172.21.15.26:0/3980271557 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0101480 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 -- 172.21.15.26:0/3980271557 >> 172.21.15.26:0/3980271557 conn(0x7f72d0077eb0 msgr2=0x7f72d00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 -- 172.21.15.26:0/3980271557 shutdown_connections 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.358+0000 7f72d5a20640 1 -- 172.21.15.26:0/3980271557 wait complete. 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 Processor -- start 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 -- start start 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0106170 0x7f72d0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 0x7f72d01a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01110b0 con 0x7f72d01010a0 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0110f30 con 0x7f72d019a020 2024-04-23T18:24:56.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72d5a20640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0111230 con 0x7f72d0106170 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37780/0 (socket says 172.21.15.26:37780) 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 -- 172.21.15.26:0/490249640 learned_addr learned my addr 172.21.15.26:0/490249640 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 0x7f72d01a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72c7fff640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0106170 0x7f72d0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 -- 172.21.15.26:0/490249640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0106170 msgr2=0x7f72d0199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0106170 0x7f72d0199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 -- 172.21.15.26:0/490249640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 msgr2=0x7f72d01a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 0x7f72d01a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72ceffd640 1 -- 172.21.15.26:0/490249640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d01a4da0 con 0x7f72d01010a0 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.359+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 0x7f72d01a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72ceffd640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f72c0005940 tx=0x7f72c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72d4a1e640 1 -- 172.21.15.26:0/490249640 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c003f030 con 0x7f72d01010a0 2024-04-23T18:24:56.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72d4a1e640 1 -- 172.21.15.26:0/490249640 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c0003cb0 con 0x7f72d01010a0 2024-04-23T18:24:56.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d01a5030 con 0x7f72d01010a0 2024-04-23T18:24:56.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72d4a1e640 1 -- 172.21.15.26:0/490249640 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0035070 con 0x7f72d01010a0 2024-04-23T18:24:56.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.360+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d01a5570 con 0x7f72d01010a0 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d4a1e640 1 -- 172.21.15.26:0/490249640 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72c0042030 con 0x7f72d01010a0 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 Processor -- start 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 -- start start 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 0x55fc23aa6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55fc239fe6d0 con 0x55fc23a99490 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fc2399b1a0 con 0x55fc23aa4480 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d880b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55fc239fa690 con 0x55fc23a99db0 2024-04-23T18:24:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37782/0 (socket says 172.21.15.26:37782) 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72cf7fe640 1 -- 172.21.15.26:0/3059841953 learned_addr learned my addr 172.21.15.26:0/3059841953 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72d4a1e640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 0x7f72bc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72c7fff640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 0x55fc23aa6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.363+0000 7f72ceffd640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 -- 172.21.15.26:0/3059841953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 msgr2=0x55fc23aa3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 -- 172.21.15.26:0/3059841953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 msgr2=0x55fc23aa6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 0x55fc23aa6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72d4a1e640 1 -- 172.21.15.26:0/490249640 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f72c00f10e0 con 0x7f72d01010a0 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 -- 172.21.15.26:0/3059841953 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc239a55b0 con 0x55fc23a99490 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72c7fff640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 0x55fc23aa6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72ceffd640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:56.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 0x7f72bc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f72c8010420 tx=0x7f72c80108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c48cf0caf01487d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72c5ffb640 1 -- 172.21.15.26:0/3059841953 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8023070 con 0x55fc23a99490 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72c5ffb640 1 -- 172.21.15.26:0/3059841953 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c801c470 con 0x55fc23a99490 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72c5ffb640 1 -- 172.21.15.26:0/3059841953 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c801d6a0 con 0x55fc23a99490 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.364+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 0x7f72bc0a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72c8020000 tx=0x7f72c801a000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- 172.21.15.26:0/3059841953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 msgr2=0x55fc23a99870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f72c8010420 tx=0x7f72c80108e0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- 172.21.15.26:0/3059841953 shutdown_connections 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23aa4480 0x55fc23aa6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 --2- 172.21.15.26:0/3059841953 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99490 0x55fc23a99870 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- 172.21.15.26:0/3059841953 >> 172.21.15.26:0/3059841953 conn(0x55fc23a957b0 msgr2=0x55fc23a95b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- 172.21.15.26:0/3059841953 shutdown_connections 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- 172.21.15.26:0/3059841953 wait complete. 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 -- start start 2024-04-23T18:24:56.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.365+0000 7f72d880b740 1 Processor -- start 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aa3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55fc23aa7310 con 0x55fc23aa4480 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fc23998b20 con 0x55fc23a99490 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55fc23aa7490 con 0x55fc23a99db0 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 10 client.0 _ll_register_callbacks cb 0x55fc23998530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37788/0 (socket says 172.21.15.26:37788) 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1016205291 learned_addr learned my addr 172.21.15.26:0/1016205291 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72c7fff640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72ceffd640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aa3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1016205291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 msgr2=0x55fc23aa1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 msgr2=0x55fc23aa3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aa3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1016205291 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc23710920 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72ceffd640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f72c801cdf0 tx=0x7f72c801d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c88c3959bd09b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8023070 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f72c801d900 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c801c470 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f72c80136c0 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.366+0000 7f72d880b740 10 client.16605 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fc237dcd30 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c801c470 con 0x55fc23aa4480 2024-04-23T18:24:56.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 39) ==== 8354+0+0 (secure 0 0 0) 0x7f72c800f040 con 0x55fc23aa4480 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 10 client.16605 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 10 client.16605 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 20 client.16605 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23ab4430 0x55fc23ab68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 4 client.16605 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc239bc500 con 0x55fc23ab4430 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.367+0000 7f72ceffd640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23ab4430 0x55fc23ab68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72ceffd640 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23ab4430 0x55fc23ab68b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93790e1f933a966b server_cookie=2c140b2aff82ebf1 in_seq=0 out_seq=0 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72aeffd640 10 client.16605 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72aeffd640 1 -- 172.21.15.26:0/1016205291 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc239bc500 con 0x55fc23ab4430 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72aeffd640 10 client.16605 handle_command_reply: tid=0 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72d880b740 1 client.16605 shutdown 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 msgr2=0x55fc23aa1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.368+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f72c801cdf0 tx=0x7f72c801d6f0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 shutdown_connections 2024-04-23T18:24:56.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23ab4430 0x55fc23ab68b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23aa1790 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99db0 0x55fc23aa1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 --2- 172.21.15.26:0/1016205291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72aeffd640 0 client.16605 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 >> 172.21.15.26:0/1016205291 conn(0x55fc23a957b0 msgr2=0x55fc23a9b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 shutdown_connections 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 1 -- 172.21.15.26:0/1016205291 wait complete. 2024-04-23T18:24:56.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.369+0000 7f72d880b740 20 client.16605 trim_cache size 0 max 16384 2024-04-23T18:24:56.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 Processor -- start 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 -- start start 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99490 0x55fc23ab01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99db0 0x55fc23aa39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01b04b0 con 0x55fc23aa4480 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01adcf0 con 0x55fc23a99db0 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72d880b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01abb60 con 0x55fc23a99490 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37798/0 (socket says 172.21.15.26:37798) 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1693764523 learned_addr learned my addr 172.21.15.26:0/1693764523 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1693764523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99490 msgr2=0x55fc23ab01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99490 0x55fc23ab01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1693764523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99db0 msgr2=0x55fc23aa39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99db0 0x55fc23aa39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.386+0000 7f72cf7fe640 1 -- 172.21.15.26:0/1693764523 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc23a9f6d0 con 0x55fc23aa4480 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72c8013570 tx=0x7f72c80097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3f07917f1a126c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72c57fa640 1 -- 172.21.15.26:0/1693764523 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c803f070 con 0x55fc23aa4480 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72c57fa640 1 -- 172.21.15.26:0/1693764523 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c801c470 con 0x55fc23aa4480 2024-04-23T18:24:56.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72c57fa640 1 -- 172.21.15.26:0/1693764523 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8023040 con 0x55fc23aa4480 2024-04-23T18:24:56.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 -- 172.21.15.26:0/1693764523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 msgr2=0x55fc23abc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72c8013570 tx=0x7f72c80097a0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 -- 172.21.15.26:0/1693764523 shutdown_connections 2024-04-23T18:24:56.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99db0 0x55fc23aa39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23aa4480 0x55fc23abc1d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 --2- 172.21.15.26:0/1693764523 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23a99490 0x55fc23ab01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 -- 172.21.15.26:0/1693764523 >> 172.21.15.26:0/1693764523 conn(0x55fc23a957b0 msgr2=0x55fc23a9abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 -- 172.21.15.26:0/1693764523 shutdown_connections 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.387+0000 7f72d880b740 1 -- 172.21.15.26:0/1693764523 wait complete. 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 -- start start 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 Processor -- start 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aacc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23aa4480 0x55fc23aa5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01ac1b0 con 0x55fc23a99db0 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01ad620 con 0x55fc23a99490 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01ac880 con 0x55fc23aa4480 2024-04-23T18:24:56.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72d880b740 10 client.0 _ll_register_callbacks cb 0x7f72c80099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72c7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72c7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37810/0 (socket says 172.21.15.26:37810) 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72c7fff640 1 -- 172.21.15.26:0/2322749172 learned_addr learned my addr 172.21.15.26:0/2322749172 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72cf7fe640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23aa4480 0x55fc23aa5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.388+0000 7f72ceffd640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aacc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 -- 172.21.15.26:0/2322749172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23aa4480 msgr2=0x55fc23aa5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23aa4480 0x55fc23aa5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 -- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 msgr2=0x55fc23aacc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aacc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 -- 172.21.15.26:0/2322749172 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc23a9f200 con 0x55fc23a99db0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72ceffd640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aacc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72c7fff640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f72d006b0a0 tx=0x7f72b8013960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424f39d935820e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:24:56.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004240 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f72b80043e0 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c801c470 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f72b80035f0 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 10 client.16611 fetch_fsmap learned FSMap version 39 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fc23b1b080 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c801c470 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 39) ==== 8354+0+0 (secure 0 0 0) 0x7f72b800f040 con 0x55fc23a99db0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 10 client.16611 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 10 client.16611 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 20 client.16611 populate_metadata read hostname 'smithi026' 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23b1b8e0 0x55fc23b1dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 4 client.16611 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1e98f5fd", "release": true, "await": true}] 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.389+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1e98f5fd", "release": true, "await": true}) -- 0x55fc23a0c0f0 con 0x55fc23b1b8e0 2024-04-23T18:24:56.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.390+0000 7f72ceffd640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23b1b8e0 0x55fc23b1dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:56.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.390+0000 7f72aeffd640 10 client.16611 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:56.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.390+0000 7f72ceffd640 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23b1b8e0 0x55fc23b1dd00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=418fe87a97912cc2 server_cookie=b24150d573416240 in_seq=0 out_seq=0 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72aeffd640 1 -- 172.21.15.26:0/2322749172 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55fc23a0c0f0 con 0x55fc23b1b8e0 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72aeffd640 10 client.16611 handle_command_reply: tid=0 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 client.16611 shutdown 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 msgr2=0x55fc23a9e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f72d006b0a0 tx=0x7f72b8013960 comp rx=0 tx=0).stop 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 shutdown_connections 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55fc23b1b8e0 0x55fc23b1dd00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fc23aa4480 0x55fc23aa5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72aeffd640 0 client.16611 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:24:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55fc23a99db0 0x55fc23a9e850 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 --2- 172.21.15.26:0/2322749172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fc23a99490 0x55fc23aacc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 >> 172.21.15.26:0/2322749172 conn(0x55fc23a957b0 msgr2=0x55fc23a9b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 shutdown_connections 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 1 -- 172.21.15.26:0/2322749172 wait complete. 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.394+0000 7f72d880b740 20 client.16611 trim_cache size 0 max 16384 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 39, 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 36, 2024-04-23T18:24:56.390 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "1e98f5fd": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 36, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 95.0, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:24:56.391 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:24:56.392 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 msgr2=0x7f72bc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 0x7f72bc0a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72c8020000 tx=0x7f72c801a000 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 msgr2=0x7f72d0199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f72c0005940 tx=0x7f72c00042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 shutdown_connections 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f72bc0a3da0 0x7f72bc0a6260 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72d019a020 0x7f72d01a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72d0106170 0x7f72d0199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 --2- 172.21.15.26:0/490249640 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f72d01010a0 0x7f72d0199550 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 >> 172.21.15.26:0/490249640 conn(0x7f72d0077eb0 msgr2=0x7f72d00feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 shutdown_connections 2024-04-23T18:24:56.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:24:56.395+0000 7f72d5a20640 1 -- 172.21.15.26:0/490249640 wait complete. 2024-04-23T18:24:56.399 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1e98f5fd', seconds elapsed: 36.2 2024-04-23T18:24:56.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 440712 5.26 MB/sec execute 291 sec latency 35592.619 ms 2024-04-23T18:24:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:57 smithi110 ceph-mon[32585]: pgmap v239: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.8 MiB/s wr, 75 op/s 2024-04-23T18:24:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:57 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/465338867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:57.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:57 smithi026 ceph-mon[29364]: pgmap v239: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.8 MiB/s wr, 75 op/s 2024-04-23T18:24:57.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:57 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/465338867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:57 smithi131 ceph-mon[33041]: pgmap v239: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 1.8 MiB/s wr, 75 op/s 2024-04-23T18:24:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:57 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/465338867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:57.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 443043 5.28 MB/sec execute 292 sec latency 17.238 ms 2024-04-23T18:24:58.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:58 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:58.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:58 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:58.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:58 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:58.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 444350 5.29 MB/sec execute 293 sec latency 23.850 ms 2024-04-23T18:24:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:24:59 smithi110 ceph-mon[32585]: pgmap v240: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:24:59.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:24:59 smithi026 ceph-mon[29364]: pgmap v240: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:24:59.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:24:59 smithi131 ceph-mon[33041]: pgmap v240: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:24:59.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 446277 5.31 MB/sec execute 294 sec latency 13.907 ms 2024-04-23T18:25:00.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 447679 5.32 MB/sec execute 295 sec latency 23.875 ms 2024-04-23T18:25:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:01 smithi110 ceph-mon[32585]: pgmap v241: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:25:01.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:01 smithi026 ceph-mon[29364]: pgmap v241: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:25:01.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:01 smithi131 ceph-mon[33041]: pgmap v241: 97 pgs: 97 active+clean; 255 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-23T18:25:01.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 448605 5.31 MB/sec execute 296 sec latency 68.017 ms 2024-04-23T18:25:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 -- 172.21.15.26:0/3439349291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a0106170 msgr2=0x7fd1a010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/3439349291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a0106170 0x7fd1a010dd40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd198009ca0 tx=0x7fd19802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 -- 172.21.15.26:0/3439349291 shutdown_connections 2024-04-23T18:25:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/3439349291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a0106170 0x7fd1a010dd40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/3439349291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0101aa0 0x7fd1a0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/3439349291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 -- 172.21.15.26:0/3439349291 >> 172.21.15.26:0/3439349291 conn(0x7fd1a0077fd0 msgr2=0x7fd1a00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 -- 172.21.15.26:0/3439349291 shutdown_connections 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.231+0000 7fd1a82a7640 1 -- 172.21.15.26:0/3439349291 wait complete. 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 Processor -- start 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 -- start start 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 0x7fd1a019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01110c0 con 0x7fd1a019d010 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0110f40 con 0x7fd1a01010f0 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a82a7640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0111240 con 0x7fd1a0106170 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 0x7fd1a019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6043640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6043640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35000/0 (socket says 172.21.15.26:35000) 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6043640 1 -- 172.21.15.26:0/1885655661 learned_addr learned my addr 172.21.15.26:0/1885655661 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 -- 172.21.15.26:0/1885655661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0106170 msgr2=0x7fd1a0198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 -- 172.21.15.26:0/1885655661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 msgr2=0x7fd1a0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.232+0000 7fd1a6844640 1 -- 172.21.15.26:0/1885655661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a019db00 con 0x7fd1a019d010 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1a6844640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 0x7fd1a019d4c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd19802ade0 tx=0x7fd1980316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1977fe640 1 -- 172.21.15.26:0/1885655661 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198003b70 con 0x7fd1a019d010 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1977fe640 1 -- 172.21.15.26:0/1885655661 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd198003d10 con 0x7fd1a019d010 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a019dd90 con 0x7fd1a019d010 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1977fe640 1 -- 172.21.15.26:0/1885655661 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19802f9e0 con 0x7fd1a019d010 2024-04-23T18:25:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.233+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a01a55b0 con 0x7fd1a019d010 2024-04-23T18:25:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.235+0000 7fd1977fe640 1 -- 172.21.15.26:0/1885655661 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd19802fb80 con 0x7fd1a019d010 2024-04-23T18:25:02.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 Processor -- start 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 -- start start 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 0x55a998fbf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fbf800 0x55a998fc1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a998f19970 con 0x55a998fbf800 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a998eb64a0 con 0x55a998fb4620 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a998f15930 con 0x55a998fb4f40 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1977fe640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 0x7fd18c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.236+0000 7fd1a6043640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 0x55a998fbf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:35014/0 (socket says 172.21.15.26:35014) 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 -- 172.21.15.26:0/4089126587 learned_addr learned my addr 172.21.15.26:0/4089126587 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fbf800 0x55a998fc1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a6844640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 0x7fd18c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1977fe640 1 -- 172.21.15.26:0/1885655661 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7fd1980f1a10 con 0x7fd1a019d010 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 -- 172.21.15.26:0/4089126587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 msgr2=0x55a998fbf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.237 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 0x55a998fbf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 -- 172.21.15.26:0/4089126587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fbf800 msgr2=0x55a998fc1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fbf800 0x55a998fc1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 -- 172.21.15.26:0/4089126587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a998c2c210 con 0x55a998fb4620 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a6043640 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 0x55a998fbf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a5842640 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd19000bfd0 tx=0x7fd19000b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e90e8339cddec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd1a6844640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 0x7fd18c0a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1980bd780 tx=0x7fd19804e000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd194bfe640 1 -- 172.21.15.26:0/4089126587 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190012c40 con 0x55a998fb4620 2024-04-23T18:25:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd194bfe640 1 -- 172.21.15.26:0/4089126587 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd190012de0 con 0x55a998fb4620 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.237+0000 7fd194bfe640 1 -- 172.21.15.26:0/4089126587 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1900047a0 con 0x55a998fb4620 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 -- 172.21.15.26:0/4089126587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 msgr2=0x55a998fb4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd19000bfd0 tx=0x7fd19000b650 comp rx=0 tx=0).stop 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 -- 172.21.15.26:0/4089126587 shutdown_connections 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fbf800 0x55a998fc1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4f40 0x55a998fbf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/4089126587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fb4620 0x55a998fb4a00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 -- 172.21.15.26:0/4089126587 >> 172.21.15.26:0/4089126587 conn(0x55a998fb0b30 msgr2=0x55a998fb0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 -- 172.21.15.26:0/4089126587 shutdown_connections 2024-04-23T18:25:02.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.238+0000 7fd1aafbf740 1 -- 172.21.15.26:0/4089126587 wait complete. 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 -- start start 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 Processor -- start 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4620 0x55a998fbf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fbc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a998fc2430 con 0x55a998fb4f40 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a998eb40b0 con 0x55a998fbf800 2024-04-23T18:25:02.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a998fc25b0 con 0x55a998fb4620 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1aafbf740 10 client.0 _ll_register_callbacks cb 0x55a998ea1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34898/0 (socket says 172.21.15.26:34898) 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 -- 172.21.15.26:0/1558860343 learned_addr learned my addr 172.21.15.26:0/1558860343 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4620 0x55a998fbf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fbc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 -- 172.21.15.26:0/1558860343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4620 msgr2=0x55a998fbf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4620 0x55a998fbf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 -- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 msgr2=0x55a998fbc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fbc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 -- 172.21.15.26:0/1558860343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a998cf8620 con 0x55a998fb4f40 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fbc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd19c012810 tx=0x7fd19c012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e129e9f588e2b2a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.239+0000 7fd189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c004050 con 0x55a998fb4f40 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7fd19c0041f0 con 0x55a998fb4f40 2024-04-23T18:25:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd190012de0 con 0x55a998fb4f40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd19c0043f0 con 0x55a998fb4f40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 10 client.16623 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a998fce400 con 0x55a998fb4f40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd190012de0 con 0x55a998fb4f40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 40) ==== 8350+0+0 (secure 0 0 0) 0x7fd19c003b80 con 0x55a998fb4f40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 10 client.16623 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:02.243 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 20 client.16623 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a998fcf980 0x55a998fd1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 4 client.16623 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a998f27390 con 0x55a998fcf980 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.240+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a998fcf980 0x55a998fd1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.241+0000 7fd189ffb640 10 client.16623 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.241+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a998fcf980 0x55a998fd1da0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c728a78962409e server_cookie=9ddaf1142aa99ae4 in_seq=0 out_seq=0 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.241+0000 7fd189ffb640 1 -- 172.21.15.26:0/1558860343 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a998f27390 con 0x55a998fcf980 2024-04-23T18:25:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.241+0000 7fd189ffb640 10 client.16623 handle_command_reply: tid=0 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.241+0000 7fd1aafbf740 1 client.16623 shutdown 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 msgr2=0x55a998fbc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd19c012810 tx=0x7fd19c012cd0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 shutdown_connections 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a998fcf980 0x55a998fd1da0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fbc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4f40 0x55a998fbc440 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd189ffb640 0 client.16623 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1558860343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fb4620 0x55a998fbf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 >> 172.21.15.26:0/1558860343 conn(0x55a998fb0b30 msgr2=0x55a998fb6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 shutdown_connections 2024-04-23T18:25:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1558860343 wait complete. 2024-04-23T18:25:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.242+0000 7fd1aafbf740 20 client.16623 trim_cache size 0 max 16384 2024-04-23T18:25:02.252 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 Processor -- start 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 -- start start 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fafb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a998fd76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01b0570 con 0x55a998fb4620 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01addb0 con 0x55a998fbf800 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01abc20 con 0x55a998fbb170 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34902/0 (socket says 172.21.15.26:34902) 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1a5842640 1 -- 172.21.15.26:0/2616417561 learned_addr learned my addr 172.21.15.26:0/2616417561 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1a6043640 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fafb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.258+0000 7fd1a5842640 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a998fd76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 -- 172.21.15.26:0/2616417561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 msgr2=0x55a998fd76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a998fd76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 -- 172.21.15.26:0/2616417561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 msgr2=0x55a998fafb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fafb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 -- 172.21.15.26:0/2616417561 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a998fbcd10 con 0x55a998fb4620 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd1a5842640 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd1900058c0 tx=0x7fd19000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7c394f993f9fab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd18bfff640 1 -- 172.21.15.26:0/2616417561 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190012d00 con 0x55a998fb4620 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd18bfff640 1 -- 172.21.15.26:0/2616417561 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd190010d90 con 0x55a998fb4620 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.259+0000 7fd18bfff640 1 -- 172.21.15.26:0/2616417561 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19002d8e0 con 0x55a998fb4620 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 -- 172.21.15.26:0/2616417561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 msgr2=0x55a998fbec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd1900058c0 tx=0x7fd19000b650 comp rx=0 tx=0).stop 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 -- 172.21.15.26:0/2616417561 shutdown_connections 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fafb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a998fd76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/2616417561 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a998fbec80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 -- 172.21.15.26:0/2616417561 >> 172.21.15.26:0/2616417561 conn(0x55a998fb0b30 msgr2=0x55a998faf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 -- 172.21.15.26:0/2616417561 shutdown_connections 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.260+0000 7fd1aafbf740 1 -- 172.21.15.26:0/2616417561 wait complete. 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 -- start start 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 Processor -- start 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a99902f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fc3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ac270 con 0x55a998fb4620 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ad6e0 con 0x55a998fbf800 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ac940 con 0x55a998fbb170 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1aafbf740 10 client.0 _ll_register_callbacks cb 0x7fd19c012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:34910/0 (socket says 172.21.15.26:34910) 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 -- 172.21.15.26:0/1856476978 learned_addr learned my addr 172.21.15.26:0/1856476978 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fc3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a99902f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 -- 172.21.15.26:0/1856476978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 msgr2=0x55a99902f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a99902f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 -- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 msgr2=0x55a998fc3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fc3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.261+0000 7fd1a5842640 1 -- 172.21.15.26:0/1856476978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a998fc4130 con 0x55a998fb4620 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1a5842640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd190011010 tx=0x7fd190005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633682d992b8cd62 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19002d560 con 0x55a998fb4620 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7fd1900108e0 con 0x55a998fb4620 2024-04-23T18:25:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd190010d90 con 0x55a998fb4620 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd190010ae0 con 0x55a998fb4620 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 10 client.16629 fetch_fsmap learned FSMap version 40 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a9990369e0 con 0x55a998fb4620 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd190010d90 con 0x55a998fb4620 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 40) ==== 8350+0+0 (secure 0 0 0) 0x7fd19001f840 con 0x55a998fb4620 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 10 client.16629 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.262+0000 7fd1aafbf740 20 client.16629 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a999036f10 0x55a999039330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd1aafbf740 4 client.16629 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a998f21200 con 0x55a999036f10 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a999036f10 0x55a999039330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd1a6043640 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a999036f10 0x55a999039330 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1533be251b423d18 server_cookie=78013655cf297a33 in_seq=0 out_seq=0 2024-04-23T18:25:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.263+0000 7fd189ffb640 10 client.16629 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.290+0000 7fd189ffb640 1 -- 172.21.15.26:0/1856476978 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a998f21200 con 0x55a999036f10 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd189ffb640 10 client.16629 handle_command_reply: tid=0 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 client.16629 shutdown 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 msgr2=0x55a99902f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd190011010 tx=0x7fd190005850 comp rx=0 tx=0).stop 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 shutdown_connections 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a999036f10 0x55a999039330 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a998fbf800 0x55a998fc3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a998fbb170 0x55a99902f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 --2- 172.21.15.26:0/1856476978 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a998fb4620 0x55a99902f230 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd189ffb640 0 client.16629 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 >> 172.21.15.26:0/1856476978 conn(0x55a998fb0b30 msgr2=0x55a998fb6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 shutdown_connections 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 1 -- 172.21.15.26:0/1856476978 wait complete. 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.291+0000 7fd1aafbf740 20 client.16629 trim_cache size 0 max 16384 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:25:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:25:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:25:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 msgr2=0x7fd18c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 0x7fd18c0a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1980bd780 tx=0x7fd19804e000 comp rx=0 tx=0).stop 2024-04-23T18:25:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 msgr2=0x7fd1a019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 0x7fd1a019d4c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd19802ade0 tx=0x7fd1980316f0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 shutdown_connections 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd18c0a3dc0 0x7fd18c0a6280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd1a019d010 0x7fd1a019d4c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 --2- 172.21.15.26:0/1885655661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 >> 172.21.15.26:0/1885655661 conn(0x7fd1a0077fd0 msgr2=0x7fd1a0069970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 shutdown_connections 2024-04-23T18:25:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:02.292+0000 7fd1a82a7640 1 -- 172.21.15.26:0/1885655661 wait complete. 2024-04-23T18:25:02.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:02 smithi110 ceph-mon[32585]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:02.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:02 smithi026 ceph-mon[29364]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:02.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:02 smithi131 ceph-mon[33041]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:02.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 450730 5.34 MB/sec execute 297 sec latency 73.376 ms 2024-04-23T18:25:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:03 smithi110 ceph-mon[32585]: pgmap v243: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:03.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:03 smithi026 ceph-mon[29364]: pgmap v243: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:03.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:03 smithi131 ceph-mon[33041]: pgmap v243: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:03.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 451931 5.34 MB/sec execute 298 sec latency 18.568 ms 2024-04-23T18:25:04.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 454062 5.37 MB/sec execute 299 sec latency 14.488 ms 2024-04-23T18:25:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:05 smithi110 ceph-mon[32585]: pgmap v244: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:05 smithi026 ceph-mon[29364]: pgmap v244: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:05.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:05 smithi131 ceph-mon[33041]: pgmap v244: 97 pgs: 97 active+clean; 260 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-23T18:25:05.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 455289 5.36 MB/sec execute 300 sec latency 22.986 ms 2024-04-23T18:25:06.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:06 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:06.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:06 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:06.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:06 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:06.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 457409 5.38 MB/sec execute 301 sec latency 14.803 ms 2024-04-23T18:25:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:07 smithi110 ceph-mon[32585]: pgmap v245: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-04-23T18:25:07.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:07 smithi110 ceph-mon[32585]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:07 smithi026 ceph-mon[29364]: pgmap v245: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-04-23T18:25:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:07 smithi026 ceph-mon[29364]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:07 smithi131 ceph-mon[33041]: pgmap v245: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.8 MiB/s wr, 263 op/s 2024-04-23T18:25:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:07 smithi131 ceph-mon[33041]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:07.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 458345 5.38 MB/sec execute 302 sec latency 23.846 ms 2024-04-23T18:25:08.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 459867 5.45 MB/sec execute 303 sec latency 50.558 ms 2024-04-23T18:25:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:09 smithi110 ceph-mon[32585]: pgmap v247: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 329 op/s 2024-04-23T18:25:09.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:09 smithi026 ceph-mon[29364]: pgmap v247: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 329 op/s 2024-04-23T18:25:09.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:09 smithi131 ceph-mon[33041]: pgmap v247: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.2 MiB/s wr, 329 op/s 2024-04-23T18:25:09.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 462208 5.48 MB/sec execute 304 sec latency 15.678 ms 2024-04-23T18:25:10.684 INFO:tasks.workunit.client.0.smithi026.stdout: 1 463598 5.48 MB/sec execute 305 sec latency 25.958 ms 2024-04-23T18:25:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:11 smithi110 ceph-mon[32585]: pgmap v248: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-04-23T18:25:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:11 smithi026 ceph-mon[29364]: pgmap v248: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-04-23T18:25:11.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:11 smithi131 ceph-mon[33041]: pgmap v248: 97 pgs: 97 active+clean; 262 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.2 MiB/s wr, 171 op/s 2024-04-23T18:25:11.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 465820 5.51 MB/sec execute 306 sec latency 29.199 ms 2024-04-23T18:25:12.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 467321 5.51 MB/sec execute 307 sec latency 25.030 ms 2024-04-23T18:25:13.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:13 smithi110 ceph-mon[32585]: pgmap v249: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:13.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:13 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:13 smithi026 ceph-mon[29364]: pgmap v249: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:13 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:13 smithi131 ceph-mon[33041]: pgmap v249: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:13 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:13.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.52 MB/sec execute 308 sec latency 515.651 ms 2024-04-23T18:25:14.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.50 MB/sec execute 309 sec latency 1515.810 ms 2024-04-23T18:25:15.384 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 -- 172.21.15.26:0/1027159895 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81010f0 msgr2=0x7ffab81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.384 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 --2- 172.21.15.26:0/1027159895 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81010f0 0x7ffab81014d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009900 tx=0x7ffab002a410 comp rx=0 tx=0).stop 2024-04-23T18:25:15.384 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 -- 172.21.15.26:0/1027159895 shutdown_connections 2024-04-23T18:25:15.384 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 --2- 172.21.15.26:0/1027159895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab8101aa0 0x7ffab8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.384 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 --2- 172.21.15.26:0/1027159895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 --2- 172.21.15.26:0/1027159895 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81010f0 0x7ffab81014d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 -- 172.21.15.26:0/1027159895 >> 172.21.15.26:0/1027159895 conn(0x7ffab8077fd0 msgr2=0x7ffab80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 -- 172.21.15.26:0/1027159895 shutdown_connections 2024-04-23T18:25:15.385 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.391+0000 7ffabcd29640 1 -- 172.21.15.26:0/1027159895 wait complete. 2024-04-23T18:25:15.386 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.392+0000 7ffabcd29640 1 Processor -- start 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.392+0000 7ffabcd29640 1 -- start start 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 0x7ffab8102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 0x7ffab81a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81110e0 con 0x7ffab81035b0 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8110f60 con 0x7ffab81010f0 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffabcd29640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8111260 con 0x7ffab8106170 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 0x7ffab81a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 0x7ffab8102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab8103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:54674/0 (socket says 172.21.15.26:54674) 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 0x7ffab8102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:54744/0 (socket says 172.21.15.26:54744) 2024-04-23T18:25:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab5d9b640 1 -- 172.21.15.26:0/670125357 learned_addr learned my addr 172.21.15.26:0/670125357 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 -- 172.21.15.26:0/670125357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 msgr2=0x7ffab8103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab8103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 -- 172.21.15.26:0/670125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 msgr2=0x7ffab8102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 0x7ffab8102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 -- 172.21.15.26:0/670125357 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab81a6fc0 con 0x7ffab81035b0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffab6d9d640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 0x7ffab81a6920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d680 tx=0x7ffaa800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.393+0000 7ffaa77fe640 1 -- 172.21.15.26:0/670125357 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017070 con 0x7ffab81035b0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.394+0000 7ffaa77fe640 1 -- 172.21.15.26:0/670125357 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa8009820 con 0x7ffab81035b0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.394+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab81a72b0 con 0x7ffab81035b0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.394+0000 7ffaa77fe640 1 -- 172.21.15.26:0/670125357 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8004ce0 con 0x7ffab81035b0 2024-04-23T18:25:15.388 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.394+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab81a7850 con 0x7ffab81035b0 2024-04-23T18:25:15.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.396+0000 7ffaa77fe640 1 -- 172.21.15.26:0/670125357 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffaa80099d0 con 0x7ffab81035b0 2024-04-23T18:25:15.389 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.396+0000 7ffabfa40740 1 Processor -- start 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 -- start start 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2a620 0x55f44fc2aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc35800 0x55f44fc37bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f44fb8f970 con 0x55f44fc2af40 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffaa77fe640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 0x7ffaa00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f44fb2c4a0 con 0x55f44fc2a620 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffabfa40740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f44fb8b930 con 0x55f44fc35800 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.391 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab6d9d640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 0x7ffaa00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52972/0 (socket says 172.21.15.26:52972) 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 -- 172.21.15.26:0/184779316 learned_addr learned my addr 172.21.15.26:0/184779316 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc35800 0x55f44fc37bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2a620 0x55f44fc2aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffaa77fe640 1 -- 172.21.15.26:0/670125357 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7ffaa80cff70 con 0x7ffab81035b0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 -- 172.21.15.26:0/184779316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc35800 msgr2=0x55f44fc37bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc35800 0x55f44fc37bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 -- 172.21.15.26:0/184779316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2a620 msgr2=0x55f44fc2aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2a620 0x55f44fc2aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.397+0000 7ffab659c640 1 -- 172.21.15.26:0/184779316 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f44f8a2210 con 0x55f44fc2af40 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.398+0000 7ffab6d9d640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 0x7ffaa00a6300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8098930 tx=0x7ffaa8000bc0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.398+0000 7ffab659c640 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009900 tx=0x7ffab0032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a4bde450cb36bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.398+0000 7ffaa4ff9640 1 -- 172.21.15.26:0/184779316 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0040070 con 0x55f44fc2af40 2024-04-23T18:25:15.392 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.398+0000 7ffaa4ff9640 1 -- 172.21.15.26:0/184779316 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab0004030 con 0x55f44fc2af40 2024-04-23T18:25:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.398+0000 7ffaa4ff9640 1 -- 172.21.15.26:0/184779316 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0035040 con 0x55f44fc2af40 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- 172.21.15.26:0/184779316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 msgr2=0x55f44fc352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009900 tx=0x7ffab0032bd0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- 172.21.15.26:0/184779316 shutdown_connections 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc35800 0x55f44fc37bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2af40 0x55f44fc352c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 --2- 172.21.15.26:0/184779316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2a620 0x55f44fc2aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- 172.21.15.26:0/184779316 >> 172.21.15.26:0/184779316 conn(0x55f44fc26b30 msgr2=0x55f44fc26f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- 172.21.15.26:0/184779316 shutdown_connections 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- 172.21.15.26:0/184779316 wait complete. 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 -- start start 2024-04-23T18:25:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 Processor -- start 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.399+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc35170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc32460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f44fc383d0 con 0x55f44fc35800 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f44fb2a0b0 con 0x55f44fc2af40 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f44fc38550 con 0x55f44fc2a620 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 10 client.0 _ll_register_callbacks cb 0x55f44fb17d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52978/0 (socket says 172.21.15.26:52978) 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1767660894 learned_addr learned my addr 172.21.15.26:0/1767660894 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc35170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab659c640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc32460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1767660894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 msgr2=0x55f44fc35170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc35170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 msgr2=0x55f44fc32460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc32460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1767660894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f44f96e620 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009270 tx=0x7ffaac009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfc126ee84ef127 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffa9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0138e0 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7ffaac013a80 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.400+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab0004030 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffaac013c80 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 10 client.16641 fetch_fsmap learned FSMap version 42 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f44fc443f0 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab0004030 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7ffaac00f040 con 0x55f44fc35800 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 10 client.16641 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:25:15.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 20 client.16641 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fc45920 0x55f44fc47da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 4 client.16641 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f44fb9d390 con 0x55f44fc45920 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.401+0000 7ffab659c640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fc45920 0x55f44fc47da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.402+0000 7ffab659c640 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fc45920 0x55f44fc47da0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61b17d82108b45a server_cookie=2d7b238f28cb2649 in_seq=0 out_seq=0 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.402+0000 7ffa9dffb640 10 client.16641 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1767660894 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f44fb9d390 con 0x55f44fc45920 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffa9dffb640 10 client.16641 handle_command_reply: tid=0 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 client.16641 shutdown 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 msgr2=0x55f44fc329c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009270 tx=0x7ffaac009f40 comp rx=0 tx=0).stop 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 shutdown_connections 2024-04-23T18:25:15.400 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fc45920 0x55f44fc47da0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc32460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc329c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffa9dffb640 0 client.16641 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1767660894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc35170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 >> 172.21.15.26:0/1767660894 conn(0x55f44fc26b30 msgr2=0x55f44fc2c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 shutdown_connections 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 1 -- 172.21.15.26:0/1767660894 wait complete. 2024-04-23T18:25:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.403+0000 7ffabfa40740 20 client.16641 trim_cache size 0 max 16384 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 Processor -- start 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 -- start start 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc41290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc31600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81b2660 con 0x55f44fc35800 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81afed0 con 0x55f44fc2af40 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.418+0000 7ffabfa40740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81add40 con 0x55f44fc2a620 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52988/0 (socket says 172.21.15.26:52988) 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 -- 172.21.15.26:0/3407681077 learned_addr learned my addr 172.21.15.26:0/3407681077 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab659c640 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc31600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc41290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 -- 172.21.15.26:0/3407681077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 msgr2=0x55f44fc41290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc41290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 -- 172.21.15.26:0/3407681077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 msgr2=0x55f44fc31600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc31600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 -- 172.21.15.26:0/3407681077 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f44fc2ad70 con 0x55f44fc35800 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009570 tx=0x7ffaac011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80efa868687635dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffa9ffff640 1 -- 172.21.15.26:0/3407681077 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0138e0 con 0x55f44fc35800 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffa9ffff640 1 -- 172.21.15.26:0/3407681077 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaac013a80 con 0x55f44fc35800 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.419+0000 7ffa9ffff640 1 -- 172.21.15.26:0/3407681077 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac00c900 con 0x55f44fc35800 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- 172.21.15.26:0/3407681077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 msgr2=0x55f44fc25b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009570 tx=0x7ffaac011f70 comp rx=0 tx=0).stop 2024-04-23T18:25:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- 172.21.15.26:0/3407681077 shutdown_connections 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc2af40 0x55f44fc31600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc35800 0x55f44fc25b30 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- 172.21.15.26:0/3407681077 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2a620 0x55f44fc41290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- 172.21.15.26:0/3407681077 >> 172.21.15.26:0/3407681077 conn(0x55f44fc26b30 msgr2=0x55f44fc25420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- 172.21.15.26:0/3407681077 shutdown_connections 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- 172.21.15.26:0/3407681077 wait complete. 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- start start 2024-04-23T18:25:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 Processor -- start 2024-04-23T18:25:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2af40 0x55f44fc3d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc35800 0x55f44fc3a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81ae390 con 0x55f44fc2a620 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81af800 con 0x55f44fc35800 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.420+0000 7ffabfa40740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ffab81aea60 con 0x55f44fc2af40 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffabfa40740 10 client.0 _ll_register_callbacks cb 0x7ffaac0113e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:52990/0 (socket says 172.21.15.26:52990) 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1793719180 learned_addr learned my addr 172.21.15.26:0/1793719180 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab659c640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2af40 0x55f44fc3d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1793719180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2af40 msgr2=0x55f44fc3d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2af40 0x55f44fc3d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc35800 msgr2=0x55f44fc3a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc35800 0x55f44fc3a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 -- 172.21.15.26:0/1793719180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f44fc3b020 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffab5d9b640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009b60 tx=0x7ffaac014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a167736c17fa53f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffa9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0138e0 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7ffaac003970 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaac013a80 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffaac014260 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffabfa40740 10 client.16647 fetch_fsmap learned FSMap version 42 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.421+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f44fcab950 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaac013a80 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7ffaac022480 con 0x55f44fc2a620 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 10 client.16647 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 20 client.16647 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fcabfb0 0x55f44fcae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 4 client.16647 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f44fb97200 con 0x55f44fcabfb0 2024-04-23T18:25:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffab659c640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fcabfb0 0x55f44fcae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffa9dffb640 10 client.16647 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.422+0000 7ffab659c640 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fcabfb0 0x55f44fcae3d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e23f0d99ab8118 server_cookie=45bf1459f0a73e28 in_seq=0 out_seq=0 2024-04-23T18:25:15.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:15 smithi110 ceph-mon[32585]: pgmap v250: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.462+0000 7ffa9dffb640 1 -- 172.21.15.26:0/1793719180 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f44fb97200 con 0x55f44fcabfb0 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.462+0000 7ffa9dffb640 10 client.16647 handle_command_reply: tid=0 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.462+0000 7ffabfa40740 1 client.16647 shutdown 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 msgr2=0x55f44fc3d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009b60 tx=0x7ffaac014f20 comp rx=0 tx=0).stop 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 shutdown_connections 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f44fcabfb0 0x55f44fcae3d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f44fc35800 0x55f44fc3a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f44fc2af40 0x55f44fc3d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 --2- 172.21.15.26:0/1793719180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f44fc2a620 0x55f44fc3d400 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffa9dffb640 0 client.16647 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 >> 172.21.15.26:0/1793719180 conn(0x55f44fc26b30 msgr2=0x55f44fc25cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 shutdown_connections 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 1 -- 172.21.15.26:0/1793719180 wait complete. 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabfa40740 20 client.16647 trim_cache size 0 max 16384 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:25:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 msgr2=0x7ffaa00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 0x7ffaa00a6300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8098930 tx=0x7ffaa8000bc0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 msgr2=0x7ffab81a6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.463+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 0x7ffab81a6920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d680 tx=0x7ffaa800db40 comp rx=0 tx=0).stop 2024-04-23T18:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 shutdown_connections 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ffaa00a3e40 0x7ffaa00a6300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ffab81035b0 0x7ffab81a6920 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffab8106170 0x7ffab8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 --2- 172.21.15.26:0/670125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffab81010f0 0x7ffab8102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 >> 172.21.15.26:0/670125357 conn(0x7ffab8077fd0 msgr2=0x7ffab8069950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 shutdown_connections 2024-04-23T18:25:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:15.464+0000 7ffabcd29640 1 -- 172.21.15.26:0/670125357 wait complete. 2024-04-23T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:15 smithi026 ceph-mon[29364]: pgmap v250: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:15.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:15 smithi131 ceph-mon[33041]: pgmap v250: 97 pgs: 97 active+clean; 271 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-23T18:25:15.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.48 MB/sec execute 310 sec latency 2515.926 ms 2024-04-23T18:25:16.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.46 MB/sec execute 311 sec latency 3516.066 ms 2024-04-23T18:25:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:17 smithi110 ceph-mon[32585]: pgmap v251: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-23T18:25:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:17 smithi026 ceph-mon[29364]: pgmap v251: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-23T18:25:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:17 smithi131 ceph-mon[33041]: pgmap v251: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-23T18:25:17.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.45 MB/sec execute 312 sec latency 4516.170 ms 2024-04-23T18:25:18.685 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.43 MB/sec execute 313 sec latency 5516.297 ms 2024-04-23T18:25:19.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:19 smithi110 ceph-mon[32585]: pgmap v252: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-23T18:25:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:19 smithi026 ceph-mon[29364]: pgmap v252: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-23T18:25:19.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:19 smithi131 ceph-mon[33041]: pgmap v252: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-23T18:25:19.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.41 MB/sec execute 314 sec latency 6516.385 ms 2024-04-23T18:25:20.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.39 MB/sec execute 315 sec latency 7516.480 ms 2024-04-23T18:25:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:21 smithi110 ceph-mon[32585]: pgmap v253: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-23T18:25:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:21 smithi026 ceph-mon[29364]: pgmap v253: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-23T18:25:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:21 smithi131 ceph-mon[33041]: pgmap v253: 97 pgs: 97 active+clean; 278 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-23T18:25:21.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 468718 5.38 MB/sec execute 316 sec latency 8516.537 ms 2024-04-23T18:25:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:22.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:22 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:25:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:22 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:25:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:22.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:22 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:25:22.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 469460 5.37 MB/sec execute 317 sec latency 8955.169 ms 2024-04-23T18:25:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:23 smithi110 ceph-mon[32585]: pgmap v254: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-23T18:25:23.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:23 smithi026 ceph-mon[29364]: pgmap v254: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-23T18:25:23.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:23 smithi131 ceph-mon[33041]: pgmap v254: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-23T18:25:23.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 471398 5.38 MB/sec execute 318 sec latency 25.785 ms 2024-04-23T18:25:24.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 473115 5.40 MB/sec execute 319 sec latency 11.428 ms 2024-04-23T18:25:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:25 smithi110 ceph-mon[32585]: pgmap v255: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 769 KiB/s wr, 76 op/s 2024-04-23T18:25:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:25 smithi026 ceph-mon[29364]: pgmap v255: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 769 KiB/s wr, 76 op/s 2024-04-23T18:25:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:25 smithi131 ceph-mon[33041]: pgmap v255: 97 pgs: 97 active+clean; 279 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 769 KiB/s wr, 76 op/s 2024-04-23T18:25:25.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 475030 5.41 MB/sec execute 320 sec latency 14.452 ms 2024-04-23T18:25:26.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 476877 5.43 MB/sec execute 321 sec latency 17.657 ms 2024-04-23T18:25:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:27 smithi110 ceph-mon[32585]: pgmap v256: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-23T18:25:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:27 smithi026 ceph-mon[29364]: pgmap v256: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-23T18:25:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:27 smithi131 ceph-mon[33041]: pgmap v256: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-23T18:25:27.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 478886 5.44 MB/sec execute 322 sec latency 15.072 ms 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 -- 172.21.15.26:0/1592625818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 msgr2=0x7f06781084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1592625818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f06781084c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0668009900 tx=0x7f066802a480 comp rx=0 tx=0).stop 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 -- 172.21.15.26:0/1592625818 shutdown_connections 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1592625818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806a780 0x7f067806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1592625818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806b120 0x7f067810edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1592625818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f06781084c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 -- 172.21.15.26:0/1592625818 >> 172.21.15.26:0/1592625818 conn(0x7f0678076c50 msgr2=0x7f0678077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 -- 172.21.15.26:0/1592625818 shutdown_connections 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.557+0000 7f067cd8a640 1 -- 172.21.15.26:0/1592625818 wait complete. 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 Processor -- start 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 -- start start 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806a780 0x7f0678101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806b120 0x7f06781017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f06781121b0 con 0x7f06781080e0 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0678112030 con 0x7f067806b120 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067cd8a640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0678112330 con 0x7f067806a780 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47020/0 (socket says 172.21.15.26:47020) 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f067659c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806a780 0x7f0678101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 -- 172.21.15.26:0/1792735901 learned_addr learned my addr 172.21.15.26:0/1792735901 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0675d9b640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806b120 0x7f06781017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 -- 172.21.15.26:0/1792735901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806a780 msgr2=0x7f0678101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806a780 0x7f0678101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 -- 172.21.15.26:0/1792735901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806b120 msgr2=0x7f06781017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806b120 0x7f06781017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.558+0000 7f0676d9d640 1 -- 172.21.15.26:0/1792735901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06781a6a70 con 0x7f06781080e0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f0676d9d640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f067000eb10 tx=0x7f067000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f06677fe640 1 -- 172.21.15.26:0/1792735901 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670004b20 con 0x7f06781080e0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06781a6d60 con 0x7f06781080e0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f06677fe640 1 -- 172.21.15.26:0/1792735901 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0670004cc0 con 0x7f06781080e0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f06677fe640 1 -- 172.21.15.26:0/1792735901 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670010690 con 0x7f06781080e0 2024-04-23T18:25:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.559+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06781a7280 con 0x7f06781080e0 2024-04-23T18:25:28.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.561+0000 7f06677fe640 1 -- 172.21.15.26:0/1792735901 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0670005020 con 0x7f06781080e0 2024-04-23T18:25:28.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 Processor -- start 2024-04-23T18:25:28.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 -- start start 2024-04-23T18:25:28.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f68296b620 0x55f68296ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.560 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 0x55f6829762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f6828d0970 con 0x55f682976800 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f68286d4a0 con 0x55f68296b620 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067fb77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f6828cc930 con 0x55f68296bf40 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f06677fe640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 0x7f06600a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f0675d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067659c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 0x55f6829762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f0675d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47024/0 (socket says 172.21.15.26:47024) 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f0675d9b640 1 -- 172.21.15.26:0/1015006880 learned_addr learned my addr 172.21.15.26:0/1015006880 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f0676d9d640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 0x7f06600a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f06677fe640 1 -- 172.21.15.26:0/1792735901 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f06700d0c60 con 0x7f06781080e0 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f0675d9b640 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f68296b620 0x55f68296ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067659c640 1 -- 172.21.15.26:0/1015006880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f68296b620 msgr2=0x55f68296ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067659c640 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f68296b620 0x55f68296ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067659c640 1 -- 172.21.15.26:0/1015006880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 msgr2=0x55f682978bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.561 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.562+0000 7f067659c640 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f067659c640 1 -- 172.21.15.26:0/1015006880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6825e3210 con 0x55f68296bf40 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f0675d9b640 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f0676d9d640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 0x7f06600a62b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06700ce460 tx=0x7f0670092090 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f067659c640 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 0x55f6829762c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0668009230 tx=0x7f0668032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6050a2df1aa8dc80 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f0664ff9640 1 -- 172.21.15.26:0/1015006880 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668040070 con 0x55f68296bf40 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f0664ff9640 1 -- 172.21.15.26:0/1015006880 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0668004030 con 0x55f68296bf40 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.563+0000 7f0664ff9640 1 -- 172.21.15.26:0/1015006880 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668035040 con 0x55f68296bf40 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- 172.21.15.26:0/1015006880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 msgr2=0x55f6829762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 0x55f6829762c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0668009230 tx=0x7f0668032b20 comp rx=0 tx=0).stop 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- 172.21.15.26:0/1015006880 shutdown_connections 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f682976800 0x55f682978bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296bf40 0x55f6829762c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 --2- 172.21.15.26:0/1015006880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f68296b620 0x55f68296ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- 172.21.15.26:0/1015006880 >> 172.21.15.26:0/1015006880 conn(0x55f682967b30 msgr2=0x55f682967f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- 172.21.15.26:0/1015006880 shutdown_connections 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- 172.21.15.26:0/1015006880 wait complete. 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 -- start start 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.564+0000 7f067fb77740 1 Processor -- start 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f6829761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682973ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f6829793d0 con 0x55f68296bf40 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f68286b0b0 con 0x55f682976800 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f682979550 con 0x55f68296b620 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 10 client.0 _ll_register_callbacks cb 0x55f682858d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47036/0 (socket says 172.21.15.26:47036) 2024-04-23T18:25:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 -- 172.21.15.26:0/78430844 learned_addr learned my addr 172.21.15.26:0/78430844 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f0675d9b640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f6829761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f0675d9b640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682973ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 -- 172.21.15.26:0/78430844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 msgr2=0x55f6829761c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f6829761c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 -- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 msgr2=0x55f682973ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682973ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 -- 172.21.15.26:0/78430844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6826af620 con 0x55f68296bf40 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067659c640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0668009430 tx=0x7f06680041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc8e5c93dab5aa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f065dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668040070 con 0x55f68296bf40 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f06680316a0 con 0x55f68296bf40 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0668004030 con 0x55f68296bf40 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0668056050 con 0x55f68296bf40 2024-04-23T18:25:28.564 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 10 client.16659 fetch_fsmap learned FSMap version 42 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.565+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f682985360 con 0x55f68296bf40 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0668004030 con 0x55f68296bf40 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f0668004c10 con 0x55f68296bf40 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 10 client.16659 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 20 client.16659 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829866b0 0x55f682988aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 4 client.16659 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6828de390 con 0x55f6829866b0 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.566+0000 7f0675d9b640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829866b0 0x55f682988aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f0675d9b640 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829866b0 0x55f682988aa0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbf340e4ce89c5c5 server_cookie=31d769efa37ebcf in_seq=0 out_seq=0 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f065dffb640 10 client.16659 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f065dffb640 1 -- 172.21.15.26:0/78430844 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6828de390 con 0x55f6829866b0 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f065dffb640 10 client.16659 handle_command_reply: tid=0 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 client.16659 shutdown 2024-04-23T18:25:28.565 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 msgr2=0x55f6829734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0668009430 tx=0x7f06680041e0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 shutdown_connections 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829866b0 0x55f682988aa0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682973ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f6829734b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f065dffb640 0 client.16659 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 --2- 172.21.15.26:0/78430844 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f6829761c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 >> 172.21.15.26:0/78430844 conn(0x55f682967b30 msgr2=0x55f68296d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 shutdown_connections 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.567+0000 7f067fb77740 1 -- 172.21.15.26:0/78430844 wait complete. 2024-04-23T18:25:28.566 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.568+0000 7f067fb77740 20 client.16659 trim_cache size 0 max 16384 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 Processor -- start 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 -- start start 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f682982290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682975cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f06781b25b0 con 0x55f68296bf40 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06781afe20 con 0x55f682976800 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067fb77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f06781adc90 con 0x55f68296b620 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47050/0 (socket says 172.21.15.26:47050) 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f067659c640 1 -- 172.21.15.26:0/800199180 learned_addr learned my addr 172.21.15.26:0/800199180 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f0675d9b640 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f682982290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.584+0000 7f0675d9b640 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682975cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.585+0000 7f067659c640 1 -- 172.21.15.26:0/800199180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 msgr2=0x55f682982290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.585+0000 7f067659c640 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f682982290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.585+0000 7f067659c640 1 -- 172.21.15.26:0/800199180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 msgr2=0x55f682975cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.585+0000 7f067659c640 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682975cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.585+0000 7f067659c640 1 -- 172.21.15.26:0/800199180 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f682962090 con 0x55f68296bf40 2024-04-23T18:25:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.586+0000 7f067659c640 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0668009230 tx=0x7f0668001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756de22c2c81e710 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.586+0000 7f065ffff640 1 -- 172.21.15.26:0/800199180 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668040070 con 0x55f68296bf40 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.586+0000 7f065ffff640 1 -- 172.21.15.26:0/800199180 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066804d070 con 0x55f68296bf40 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.586+0000 7f065ffff640 1 -- 172.21.15.26:0/800199180 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668035040 con 0x55f68296bf40 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- 172.21.15.26:0/800199180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 msgr2=0x55f68296e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0668009230 tx=0x7f0668001e70 comp rx=0 tx=0).stop 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- 172.21.15.26:0/800199180 shutdown_connections 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682975cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296e970 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- 172.21.15.26:0/800199180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f682982290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- 172.21.15.26:0/800199180 >> 172.21.15.26:0/800199180 conn(0x55f682967b30 msgr2=0x55f68296e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- 172.21.15.26:0/800199180 shutdown_connections 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- 172.21.15.26:0/800199180 wait complete. 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 -- start start 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 Processor -- start 2024-04-23T18:25:28.583 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f68297e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.587+0000 7f067fb77740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682970280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f06781ae2e0 con 0x55f68296bf40 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06781af750 con 0x55f682976800 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f06781ae9b0 con 0x55f68296b620 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 10 client.0 _ll_register_callbacks cb 0x7f06680095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f0675d9b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f68297e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:47064/0 (socket says 172.21.15.26:47064) 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 -- 172.21.15.26:0/3630255517 learned_addr learned my addr 172.21.15.26:0/3630255517 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 -- 172.21.15.26:0/3630255517 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 msgr2=0x55f68297e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f68297e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f0675d9b640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682970280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 -- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 msgr2=0x55f682970280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682970280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 -- 172.21.15.26:0/3630255517 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f682974450 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067659c640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0668046040 tx=0x7f0668031dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbd055932616d10 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f065dffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668035040 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f0668005200 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f066804d070 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0668040040 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 10 client.16665 fetch_fsmap learned FSMap version 42 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.588+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f6829ec520 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066804d070 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 42) ==== 8350+0+0 (secure 0 0 0) 0x7f06680037b0 con 0x55f68296bf40 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 10 client.16665 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 10 client.16665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 20 client.16665 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829ecb60 0x55f6829eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 4 client.16665 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6828d8200 con 0x55f6829ecb60 2024-04-23T18:25:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f0675d9b640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829ecb60 0x55f6829eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f065dffb640 10 client.16665 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.589+0000 7f0675d9b640 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829ecb60 0x55f6829eef80 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946f7bb9f48c5591 server_cookie=f8f29e3a9d1dbf8d in_seq=0 out_seq=0 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.619+0000 7f065dffb640 1 -- 172.21.15.26:0/3630255517 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6828d8200 con 0x55f6829ecb60 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.619+0000 7f065dffb640 10 client.16665 handle_command_reply: tid=0 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.619+0000 7f067fb77740 1 client.16665 shutdown 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.619+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 msgr2=0x55f68296fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.619+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0668046040 tx=0x7f0668031dc0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 shutdown_connections 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f6829ecb60 0x55f6829eef80 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f682976800 0x55f682970280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f68296bf40 0x55f68296fd40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 --2- 172.21.15.26:0/3630255517 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f68296b620 0x55f68297e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 >> 172.21.15.26:0/3630255517 conn(0x55f682967b30 msgr2=0x55f68296d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f065dffb640 0 client.16665 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 shutdown_connections 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 1 -- 172.21.15.26:0/3630255517 wait complete. 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.620+0000 7f067fb77740 20 client.16665 trim_cache size 0 max 16384 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:25:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 msgr2=0x7f06600a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 0x7f06600a62b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06700ce460 tx=0x7f0670092090 comp rx=0 tx=0).stop 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 msgr2=0x7f0678101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f067000eb10 tx=0x7f067000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 shutdown_connections 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f06600a3df0 0x7f06600a62b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f067806b120 0x7f06781017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f06781080e0 0x7f0678101cf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 --2- 172.21.15.26:0/1792735901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f067806a780 0x7f0678101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 >> 172.21.15.26:0/1792735901 conn(0x7f0678076c50 msgr2=0x7f06780ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 shutdown_connections 2024-04-23T18:25:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:28.621+0000 7f067cd8a640 1 -- 172.21.15.26:0/1792735901 wait complete. 2024-04-23T18:25:28.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 480459 5.45 MB/sec execute 323 sec latency 17.018 ms 2024-04-23T18:25:29.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:29 smithi110 ceph-mon[32585]: pgmap v257: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:29.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:29 smithi026 ceph-mon[29364]: pgmap v257: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:29.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:29 smithi131 ceph-mon[33041]: pgmap v257: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:29.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 482069 5.46 MB/sec execute 324 sec latency 232.212 ms 2024-04-23T18:25:30.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 483811 5.48 MB/sec execute 325 sec latency 13.376 ms 2024-04-23T18:25:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:31 smithi110 ceph-mon[32585]: pgmap v258: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:31.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:31 smithi026 ceph-mon[29364]: pgmap v258: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:31 smithi131 ceph-mon[33041]: pgmap v258: 97 pgs: 97 active+clean; 284 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 664 KiB/s wr, 70 op/s 2024-04-23T18:25:31.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 485674 5.48 MB/sec execute 326 sec latency 17.169 ms 2024-04-23T18:25:32.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:32 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:32.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:32 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:32.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 487380 5.50 MB/sec execute 327 sec latency 18.820 ms 2024-04-23T18:25:32.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:32 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:33 smithi026 ceph-mon[29364]: pgmap v259: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-23T18:25:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:33 smithi131 ceph-mon[33041]: pgmap v259: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-23T18:25:33.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 489240 5.51 MB/sec execute 328 sec latency 25.002 ms 2024-04-23T18:25:33.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:33 smithi110 ceph-mon[32585]: pgmap v259: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-23T18:25:34.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 491030 5.53 MB/sec execute 329 sec latency 17.881 ms 2024-04-23T18:25:34.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:34 smithi110 ceph-mon[32585]: pgmap v260: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.4 MiB/s wr, 170 op/s 2024-04-23T18:25:34.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:34 smithi026 ceph-mon[29364]: pgmap v260: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.4 MiB/s wr, 170 op/s 2024-04-23T18:25:34.741 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:34 smithi131 ceph-mon[33041]: pgmap v260: 97 pgs: 97 active+clean; 292 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.4 MiB/s wr, 170 op/s 2024-04-23T18:25:35.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 492905 5.54 MB/sec execute 330 sec latency 20.994 ms 2024-04-23T18:25:36.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 494725 5.55 MB/sec execute 331 sec latency 13.785 ms 2024-04-23T18:25:36.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:36 smithi110 ceph-mon[32585]: pgmap v261: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-23T18:25:36.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:36 smithi026 ceph-mon[29364]: pgmap v261: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-23T18:25:36.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:36 smithi131 ceph-mon[33041]: pgmap v261: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2024-04-23T18:25:37.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 496791 5.57 MB/sec execute 332 sec latency 21.039 ms 2024-04-23T18:25:37.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:37 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:37.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:37 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:37.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:37 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:38.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 498368 5.58 MB/sec execute 333 sec latency 20.322 ms 2024-04-23T18:25:39.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:38 smithi110 ceph-mon[32585]: pgmap v262: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:39.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:38 smithi026 ceph-mon[29364]: pgmap v262: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:39.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:38 smithi131 ceph-mon[33041]: pgmap v262: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:39.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 500749 5.60 MB/sec execute 334 sec latency 14.439 ms 2024-04-23T18:25:40.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 502105 5.61 MB/sec execute 335 sec latency 40.506 ms 2024-04-23T18:25:41.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:40 smithi110 ceph-mon[32585]: pgmap v263: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:40 smithi026 ceph-mon[29364]: pgmap v263: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:41.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:40 smithi131 ceph-mon[33041]: pgmap v263: 97 pgs: 97 active+clean; 300 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-04-23T18:25:41.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 504621 5.64 MB/sec execute 336 sec latency 17.482 ms 2024-04-23T18:25:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 -- 172.21.15.26:0/3310906710 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df0106170 msgr2=0x7f3df010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 --2- 172.21.15.26:0/3310906710 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df0106170 0x7f3df010dd40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009ca0 tx=0x7f3de802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 -- 172.21.15.26:0/3310906710 shutdown_connections 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 --2- 172.21.15.26:0/3310906710 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df0106170 0x7f3df010dd40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 --2- 172.21.15.26:0/3310906710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0101aa0 0x7f3df0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 --2- 172.21.15.26:0/3310906710 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df01010f0 0x7f3df01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 -- 172.21.15.26:0/3310906710 >> 172.21.15.26:0/3310906710 conn(0x7f3df0077fd0 msgr2=0x7f3df00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 -- 172.21.15.26:0/3310906710 shutdown_connections 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.712+0000 7f3df71fe640 1 -- 172.21.15.26:0/3310906710 wait complete. 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 Processor -- start 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 -- start start 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0106170 0x7f3df0198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df019d010 0x7f3df019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01110c0 con 0x7f3df01010f0 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df0110f40 con 0x7f3df019d010 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df71fe640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3df0111240 con 0x7f3df0106170 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55222/0 (socket says 172.21.15.26:55222) 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df4f9a640 1 -- 172.21.15.26:0/1907914109 learned_addr learned my addr 172.21.15.26:0/1907914109 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3df579b640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df019d010 0x7f3df019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.713+0000 7f3deffff640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0106170 0x7f3df0198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 -- 172.21.15.26:0/1907914109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0106170 msgr2=0x7f3df0198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0106170 0x7f3df0198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 -- 172.21.15.26:0/1907914109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df019d010 msgr2=0x7f3df019d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df019d010 0x7f3df019d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 -- 172.21.15.26:0/1907914109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df019db00 con 0x7f3df01010f0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3de400e9f0 tx=0x7f3de400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3dedffb640 1 -- 172.21.15.26:0/1907914109 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de400cd80 con 0x7f3df01010f0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3dedffb640 1 -- 172.21.15.26:0/1907914109 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3de40108e0 con 0x7f3df01010f0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df019ddf0 con 0x7f3df01010f0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3dedffb640 1 -- 172.21.15.26:0/1907914109 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4018980 con 0x7f3df01010f0 2024-04-23T18:25:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.714+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df01a55e0 con 0x7f3df01010f0 2024-04-23T18:25:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3dedffb640 1 -- 172.21.15.26:0/1907914109 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3de4010430 con 0x7f3df01010f0 2024-04-23T18:25:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 Processor -- start 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 -- start start 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3dedffb640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 0x7f3de00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e22222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2222800 0x5652e2224bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5652e217c970 con 0x5652e2217620 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5652e21194a0 con 0x5652e2222800 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df9fed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5652e2178930 con 0x5652e2217f40 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df579b640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 0x7f3de00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55226/0 (socket says 172.21.15.26:55226) 2024-04-23T18:25:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.717+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2290934792 learned_addr learned my addr 172.21.15.26:0/2290934792 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df579b640 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2222800 0x5652e2224bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3dedffb640 1 -- 172.21.15.26:0/1907914109 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f3de40d0ae0 con 0x7f3df01010f0 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3deffff640 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e22222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2290934792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 msgr2=0x5652e22222c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e22222c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2290934792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2222800 msgr2=0x5652e2224bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2222800 0x5652e2224bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2290934792 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e1e8f210 con 0x5652e2217620 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3de40a51c0 tx=0x7f3de401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c44b76ed5bd3eec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3df579b640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 0x7f3de00a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3de802f500 tx=0x7f3de802f310 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3ddaffd640 1 -- 172.21.15.26:0/2290934792 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de406e8e0 con 0x5652e2217620 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3ddaffd640 1 -- 172.21.15.26:0/2290934792 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3de4010720 con 0x5652e2217620 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.718+0000 7f3ddaffd640 1 -- 172.21.15.26:0/2290934792 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de40a56f0 con 0x5652e2217620 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- 172.21.15.26:0/2290934792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 msgr2=0x5652e2217a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3de40a51c0 tx=0x7f3de401f000 comp rx=0 tx=0).stop 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- 172.21.15.26:0/2290934792 shutdown_connections 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2222800 0x5652e2224bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e22222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2290934792 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e2217a00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- 172.21.15.26:0/2290934792 >> 172.21.15.26:0/2290934792 conn(0x5652e2213b30 msgr2=0x5652e2213f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- 172.21.15.26:0/2290934792 shutdown_connections 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- 172.21.15.26:0/2290934792 wait complete. 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 -- start start 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.719+0000 7f3df9fed740 1 Processor -- start 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217f40 0x5652e221f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e221fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5652e2225370 con 0x5652e2217620 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5652e21170b0 con 0x5652e2217f40 2024-04-23T18:25:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5652e22254f0 con 0x5652e2222800 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55238/0 (socket says 172.21.15.26:55238) 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df9fed740 10 client.0 _ll_register_callbacks cb 0x5652e2104d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2273104827 learned_addr learned my addr 172.21.15.26:0/2273104827 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3deffff640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217f40 0x5652e221f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df579b640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e221fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2273104827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 msgr2=0x5652e221fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e221fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217f40 msgr2=0x5652e221f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217f40 0x5652e221f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 -- 172.21.15.26:0/2273104827 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e1f5b620 con 0x5652e2217620 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3de4098f90 tx=0x7f3de40a0ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecd38e44b70b6fb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.720+0000 7f3dcffff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de409c180 con 0x5652e2217620 2024-04-23T18:25:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f3de4018b20 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5652e218a390 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3de406ead0 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 10 client.16677 fetch_fsmap learned FSMap version 44 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5652e2231300 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5652e218a390 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f3de409b9f0 con 0x5652e2217620 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 10 client.16677 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 20 client.16677 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e2232650 0x5652e2234a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 4 client.16677 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652e2184200 con 0x5652e2232650 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.721+0000 7f3deffff640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e2232650 0x5652e2234a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.722+0000 7f3deffff640 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e2232650 0x5652e2234a40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a8fc4c45f6dc2b server_cookie=14322c68033a646e in_seq=0 out_seq=0 2024-04-23T18:25:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.722+0000 7f3dcffff640 10 client.16677 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.722+0000 7f3dcffff640 1 -- 172.21.15.26:0/2273104827 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652e2184200 con 0x5652e2232650 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.722+0000 7f3dcffff640 10 client.16677 handle_command_reply: tid=0 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.722+0000 7f3df9fed740 1 client.16677 shutdown 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 msgr2=0x5652e22221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3de4098f90 tx=0x7f3de40a0ea0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 shutdown_connections 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e2232650 0x5652e2234a40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217f40 0x5652e221f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e221fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3dcffff640 0 client.16677 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 --2- 172.21.15.26:0/2273104827 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217620 0x5652e22221c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 >> 172.21.15.26:0/2273104827 conn(0x5652e2213b30 msgr2=0x5652e22198b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 shutdown_connections 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 1 -- 172.21.15.26:0/2273104827 wait complete. 2024-04-23T18:25:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.723+0000 7f3df9fed740 20 client.16677 trim_cache size 0 max 16384 2024-04-23T18:25:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.738+0000 7f3df9fed740 1 Processor -- start 2024-04-23T18:25:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.738+0000 7f3df9fed740 1 -- start start 2024-04-23T18:25:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e222e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e223a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01b0390 con 0x5652e2217f40 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01adc60 con 0x5652e2217620 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df9fed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01abad0 con 0x5652e2222800 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55240/0 (socket says 172.21.15.26:55240) 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 -- 172.21.15.26:0/1979130169 learned_addr learned my addr 172.21.15.26:0/1979130169 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e222e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.734 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3df579b640 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e223a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 -- 172.21.15.26:0/1979130169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 msgr2=0x5652e223a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e223a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 -- 172.21.15.26:0/1979130169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 msgr2=0x5652e222e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e222e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 -- 172.21.15.26:0/1979130169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e2220420 con 0x5652e2217f40 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3deffff640 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc004700 tx=0x7f3ddc005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac36b317c5a1d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3dda7fc640 1 -- 172.21.15.26:0/1979130169 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003d20 con 0x5652e2217f40 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3dda7fc640 1 -- 172.21.15.26:0/1979130169 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ddc01d8e0 con 0x5652e2217f40 2024-04-23T18:25:41.735 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.739+0000 7f3dda7fc640 1 -- 172.21.15.26:0/1979130169 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc004a20 con 0x5652e2217f40 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- 172.21.15.26:0/1979130169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 msgr2=0x5652e2221cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc004700 tx=0x7f3ddc005b80 comp rx=0 tx=0).stop 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- 172.21.15.26:0/1979130169 shutdown_connections 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2222800 0x5652e223a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2217f40 0x5652e2221cf0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 --2- 172.21.15.26:0/1979130169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e222e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- 172.21.15.26:0/1979130169 >> 172.21.15.26:0/1979130169 conn(0x5652e2213b30 msgr2=0x5652e2212cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- 172.21.15.26:0/1979130169 shutdown_connections 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- 172.21.15.26:0/1979130169 wait complete. 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.740+0000 7f3df9fed740 1 -- start start 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 Processor -- start 2024-04-23T18:25:41.739 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e221caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01ac120 con 0x5652e2222800 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01ad590 con 0x5652e2217620 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3df01ac7f0 con 0x5652e2217f40 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df9fed740 10 client.0 _ll_register_callbacks cb 0x7f3de4099430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:55246/0 (socket says 172.21.15.26:55246) 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df4f9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:51480/0 (socket says 172.21.15.26:51480) 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 -- 172.21.15.26:0/3653937037 learned_addr learned my addr 172.21.15.26:0/3653937037 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3deffff640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e221caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 -- 172.21.15.26:0/3653937037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 msgr2=0x5652e221caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.740 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e221caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 -- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 msgr2=0x5652e221c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df579b640 1 -- 172.21.15.26:0/3653937037 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e221d370 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.741+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3df579b640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009ca0 tx=0x7f3de802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd42d6b8e9369265 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8003d00 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f3de8047040 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ddc01d8e0 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3de8004040 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3df9fed740 10 client.16683 fetch_fsmap learned FSMap version 44 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5652e229d340 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ddc01d8e0 con 0x5652e2222800 2024-04-23T18:25:41.741 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.742+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f3de804b070 con 0x5652e2222800 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 10 client.16683 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 20 client.16683 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e229daa0 0x5652e229fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 4 client.16683 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652e21005f0 con 0x5652e229daa0 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e229daa0 0x5652e229fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3df4f9a640 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e229daa0 0x5652e229fec0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a1ddb4173f83a server_cookie=1253f6e3439c1cb7 in_seq=0 out_seq=0 2024-04-23T18:25:41.742 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.743+0000 7f3dcffff640 10 client.16683 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3dcffff640 1 -- 172.21.15.26:0/3653937037 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5652e21005f0 con 0x5652e229daa0 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3dcffff640 10 client.16683 handle_command_reply: tid=0 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 client.16683 shutdown 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 msgr2=0x5652e229bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009ca0 tx=0x7f3de802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 shutdown_connections 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5652e229daa0 0x5652e229fec0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5652e2222800 0x5652e229bcf0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5652e2217f40 0x5652e221caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3dcffff640 0 client.16683 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 --2- 172.21.15.26:0/3653937037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5652e2217620 0x5652e221c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 >> 172.21.15.26:0/3653937037 conn(0x5652e2213b30 msgr2=0x5652e2212420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 shutdown_connections 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 1 -- 172.21.15.26:0/3653937037 wait complete. 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.776+0000 7f3df9fed740 20 client.16683 trim_cache size 0 max 16384 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:25:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:25:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:25:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 msgr2=0x7f3de00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 0x7f3de00a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3de802f500 tx=0x7f3de802f310 comp rx=0 tx=0).stop 2024-04-23T18:25:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 msgr2=0x7f3df0198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3de400e9f0 tx=0x7f3de400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 shutdown_connections 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3de00a3dc0 0x7f3de00a6280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df019d010 0x7f3df019d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3df0106170 0x7f3df0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 --2- 172.21.15.26:0/1907914109 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3df01010f0 0x7f3df0198750 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 >> 172.21.15.26:0/1907914109 conn(0x7f3df0077fd0 msgr2=0x7f3df0069970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 shutdown_connections 2024-04-23T18:25:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:41.777+0000 7f3df71fe640 1 -- 172.21.15.26:0/1907914109 wait complete. 2024-04-23T18:25:42.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 505926 5.64 MB/sec execute 337 sec latency 68.706 ms 2024-04-23T18:25:43.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:42 smithi110 ceph-mon[32585]: pgmap v264: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-04-23T18:25:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:42 smithi026 ceph-mon[29364]: pgmap v264: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-04-23T18:25:43.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:42 smithi131 ceph-mon[33041]: pgmap v264: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 319 op/s 2024-04-23T18:25:43.686 INFO:tasks.workunit.client.0.smithi026.stdout: 1 508151 5.66 MB/sec execute 338 sec latency 81.737 ms 2024-04-23T18:25:44.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 509469 5.66 MB/sec execute 339 sec latency 27.332 ms 2024-04-23T18:25:45.043 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:44 smithi131 ceph-mon[33041]: pgmap v265: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 212 op/s 2024-04-23T18:25:45.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:44 smithi110 ceph-mon[32585]: pgmap v265: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 212 op/s 2024-04-23T18:25:45.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:44 smithi026 ceph-mon[29364]: pgmap v265: 97 pgs: 97 active+clean; 308 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 212 op/s 2024-04-23T18:25:45.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 511795 5.69 MB/sec execute 340 sec latency 15.747 ms 2024-04-23T18:25:46.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 513089 5.68 MB/sec execute 341 sec latency 20.070 ms 2024-04-23T18:25:47.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:46 smithi110 ceph-mon[32585]: pgmap v266: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.7 MiB/s wr, 345 op/s 2024-04-23T18:25:47.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:46 smithi026 ceph-mon[29364]: pgmap v266: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.7 MiB/s wr, 345 op/s 2024-04-23T18:25:47.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:46 smithi131 ceph-mon[33041]: pgmap v266: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.7 MiB/s wr, 345 op/s 2024-04-23T18:25:47.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 515353 5.71 MB/sec execute 342 sec latency 15.305 ms 2024-04-23T18:25:48.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 516654 5.71 MB/sec execute 343 sec latency 13.957 ms 2024-04-23T18:25:49.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:48 smithi110 ceph-mon[32585]: pgmap v267: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:49.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:48 smithi026 ceph-mon[29364]: pgmap v267: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:49.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:48 smithi131 ceph-mon[33041]: pgmap v267: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:49.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 518951 5.73 MB/sec execute 344 sec latency 16.492 ms 2024-04-23T18:25:50.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 520232 5.73 MB/sec execute 345 sec latency 62.213 ms 2024-04-23T18:25:51.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:50 smithi110 ceph-mon[32585]: pgmap v268: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:50 smithi026 ceph-mon[29364]: pgmap v268: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:51.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:50 smithi131 ceph-mon[33041]: pgmap v268: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 240 op/s 2024-04-23T18:25:51.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 522502 5.75 MB/sec execute 346 sec latency 14.024 ms 2024-04-23T18:25:52.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 523918 5.75 MB/sec execute 347 sec latency 20.114 ms 2024-04-23T18:25:53.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:52 smithi110 ceph-mon[32585]: pgmap v269: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-23T18:25:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:52 smithi026 ceph-mon[29364]: pgmap v269: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-23T18:25:53.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:52 smithi131 ceph-mon[33041]: pgmap v269: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-23T18:25:53.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 526125 5.78 MB/sec execute 348 sec latency 12.799 ms 2024-04-23T18:25:54.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 527743 5.77 MB/sec execute 349 sec latency 16.192 ms 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 -- 172.21.15.26:0/1441473363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d810c550 msgr2=0x7f93d810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 --2- 172.21.15.26:0/1441473363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d810c550 0x7f93d810e9d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009ca0 tx=0x7f93d002a950 comp rx=0 tx=0).stop 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 -- 172.21.15.26:0/1441473363 shutdown_connections 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 --2- 172.21.15.26:0/1441473363 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d810c550 0x7f93d810e9d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 --2- 172.21.15.26:0/1441473363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 0x7f93d810bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 --2- 172.21.15.26:0/1441473363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d8103180 0x7f93d8103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 -- 172.21.15.26:0/1441473363 >> 172.21.15.26:0/1441473363 conn(0x7f93d8077fd0 msgr2=0x7f93d80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 -- 172.21.15.26:0/1441473363 shutdown_connections 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.874+0000 7f93dec72640 1 -- 172.21.15.26:0/1441473363 wait complete. 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 Processor -- start 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 -- start start 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 0x7f93d8101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 0x7f93d8101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81121c0 con 0x7f93d8103180 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8112040 con 0x7f93d810c550 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dec72640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8112340 con 0x7f93d8103aa0 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53204/0 (socket says 172.21.15.26:53204) 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 -- 172.21.15.26:0/1279762469 learned_addr learned my addr 172.21.15.26:0/1279762469 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93d7fff640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 0x7f93d8101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dd20f640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 0x7f93d8101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 -- 172.21.15.26:0/1279762469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 msgr2=0x7f93d8101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 0x7f93d8101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 -- 172.21.15.26:0/1279762469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 msgr2=0x7f93d8101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 0x7f93d8101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.875+0000 7f93dca0e640 1 -- 172.21.15.26:0/1279762469 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d81a6b00 con 0x7f93d8103180 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93dd20f640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 0x7f93d8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93dca0e640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9f0 tx=0x7f93c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93d5ffb640 1 -- 172.21.15.26:0/1279762469 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c800cd80 con 0x7f93d8103180 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93d5ffb640 1 -- 172.21.15.26:0/1279762469 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c80108e0 con 0x7f93d8103180 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93d5ffb640 1 -- 172.21.15.26:0/1279762469 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8018980 con 0x7f93d8103180 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d81a6df0 con 0x7f93d8103180 2024-04-23T18:25:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.876+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d81a7300 con 0x7f93d8103180 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.878+0000 7f93d5ffb640 1 -- 172.21.15.26:0/1279762469 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f93c8010430 con 0x7f93d8103180 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 Processor -- start 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 -- start start 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb086620 0x559fcb086a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 0x559fcb093bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x559fcafeb970 con 0x559fcb086620 2024-04-23T18:25:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559fcaf884a0 con 0x559fcb091800 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93e1987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x559fcafe7930 con 0x559fcb086f40 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93d5ffb640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 0x7f93c00a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93dd20f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb086620 0x559fcb086a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 0x559fcb093bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:52778/0 (socket says 172.21.15.26:52778) 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.879+0000 7f93dca0e640 1 -- 172.21.15.26:0/2309897381 learned_addr learned my addr 172.21.15.26:0/2309897381 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d5ffb640 1 -- 172.21.15.26:0/1279762469 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f93c80d0a60 con 0x7f93d8103180 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93dd20f640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 0x7f93c00a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 -- 172.21.15.26:0/2309897381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 msgr2=0x559fcb0912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 -- 172.21.15.26:0/2309897381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb086620 msgr2=0x559fcb086a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb086620 0x559fcb086a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 -- 172.21.15.26:0/2309897381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fcacfe210 con 0x559fcb091800 2024-04-23T18:25:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93dca0e640 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93d7fff640 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 0x559fcb093bf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93cc005fd0 tx=0x7f93cc00ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b12f3028608b0b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93beffd640 1 -- 172.21.15.26:0/2309897381 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc003a90 con 0x559fcb091800 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93beffd640 1 -- 172.21.15.26:0/2309897381 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93cc015070 con 0x559fcb091800 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93beffd640 1 -- 172.21.15.26:0/2309897381 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc004bf0 con 0x559fcb091800 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.880+0000 7f93dd20f640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 0x7f93c00a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009430 tx=0x7f93d0005c50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 -- 172.21.15.26:0/2309897381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 msgr2=0x559fcb093bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 0x559fcb093bf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93cc005fd0 tx=0x7f93cc00ba60 comp rx=0 tx=0).stop 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 -- 172.21.15.26:0/2309897381 shutdown_connections 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb091800 0x559fcb093bf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 --2- 172.21.15.26:0/2309897381 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb086620 0x559fcb086a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 -- 172.21.15.26:0/2309897381 >> 172.21.15.26:0/2309897381 conn(0x559fcb082b30 msgr2=0x559fcb082f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 -- 172.21.15.26:0/2309897381 shutdown_connections 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.881+0000 7f93e1987740 1 -- 172.21.15.26:0/2309897381 wait complete. 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 -- start start 2024-04-23T18:25:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 Processor -- start 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb0910f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb08e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x559fcb094430 con 0x559fcb091800 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559fcaf860b0 con 0x559fcb086f40 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x559fcb0945b0 con 0x559fcb086620 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93e1987740 10 client.0 _ll_register_callbacks cb 0x559fcaf73d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93dd20f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93dd20f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53226/0 (socket says 172.21.15.26:53226) 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93dd20f640 1 -- 172.21.15.26:0/3261450352 learned_addr learned my addr 172.21.15.26:0/3261450352 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93dca0e640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb0910f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.882+0000 7f93d7fff640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb08e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 -- 172.21.15.26:0/3261450352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 msgr2=0x559fcb0910f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb0910f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 -- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 msgr2=0x559fcb08e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb08e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 -- 172.21.15.26:0/3261450352 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fcadca620 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93dd20f640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f93d0035000 tx=0x7f93d00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dd3a0c12a7938e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d004f030 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f93d00047d0 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93cc015070 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f93d0034040 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 10 client.16695 fetch_fsmap learned FSMap version 44 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559fcb0a0300 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93cc015070 con 0x559fcb091800 2024-04-23T18:25:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f93d004ba30 con 0x559fcb091800 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 10 client.16695 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 10 client.16695 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 20 client.16695 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb0a17c0 0x559fcb0a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 4 client.16695 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.883+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fcaff9390 con 0x559fcb0a17c0 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.884+0000 7f93dca0e640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb0a17c0 0x559fcb0a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.884+0000 7f93dca0e640 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb0a17c0 0x559fcb0a3bb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f89ca1d8041c2196 server_cookie=fde2bd326a5f5ea8 in_seq=0 out_seq=0 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.884+0000 7f93b7fff640 10 client.16695 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93b7fff640 1 -- 172.21.15.26:0/3261450352 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559fcaff9390 con 0x559fcb0a17c0 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93b7fff640 10 client.16695 handle_command_reply: tid=0 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 client.16695 shutdown 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 msgr2=0x559fcb08e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f93d0035000 tx=0x7f93d00095c0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 shutdown_connections 2024-04-23T18:25:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb0a17c0 0x559fcb0a3bb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb08e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb091800 0x559fcb08e920 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 --2- 172.21.15.26:0/3261450352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb0910f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 >> 172.21.15.26:0/3261450352 conn(0x559fcb082b30 msgr2=0x559fcb0888b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93b7fff640 0 client.16695 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 shutdown_connections 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 1 -- 172.21.15.26:0/3261450352 wait complete. 2024-04-23T18:25:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.885+0000 7f93e1987740 20 client.16695 trim_cache size 0 max 16384 2024-04-23T18:25:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.901+0000 7f93e1987740 1 Processor -- start 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.901+0000 7f93e1987740 1 -- start start 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 0x559fcb0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81b2540 con 0x559fcb082b30 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81afe10 con 0x559fcb086620 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93e1987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81adc80 con 0x559fcb086f40 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 0x559fcb0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53242/0 (socket says 172.21.15.26:53242) 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dd20f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dca0e640 1 -- 172.21.15.26:0/3247924283 learned_addr learned my addr 172.21.15.26:0/3247924283 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 -- 172.21.15.26:0/3247924283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 msgr2=0x559fcb0a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 -- 172.21.15.26:0/3247924283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 msgr2=0x559fcb09d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 -- 172.21.15.26:0/3247924283 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fcb089fe0 con 0x559fcb086620 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dd20f640 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93dca0e640 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.902+0000 7f93d7fff640 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 0x559fcb0a94c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00b5f0 tx=0x7f93cc0038d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31184e1ba12d433 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93be7fc640 1 -- 172.21.15.26:0/3247924283 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc029070 con 0x559fcb086620 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93be7fc640 1 -- 172.21.15.26:0/3247924283 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93cc015050 con 0x559fcb086620 2024-04-23T18:25:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93be7fc640 1 -- 172.21.15.26:0/3247924283 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc011440 con 0x559fcb086620 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 -- 172.21.15.26:0/3247924283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 msgr2=0x559fcb0a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 0x559fcb0a94c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00b5f0 tx=0x7f93cc0038d0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 -- 172.21.15.26:0/3247924283 shutdown_connections 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086620 0x559fcb0a94c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086f40 0x559fcb0a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 --2- 172.21.15.26:0/3247924283 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb09d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 -- 172.21.15.26:0/3247924283 >> 172.21.15.26:0/3247924283 conn(0x559fcb08ea80 msgr2=0x559fcb081420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 -- 172.21.15.26:0/3247924283 shutdown_connections 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.903+0000 7f93e1987740 1 -- 172.21.15.26:0/3247924283 wait complete. 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 -- start start 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 Processor -- start 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb096f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81ae2d0 con 0x559fcb082b30 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81af740 con 0x559fcb086f40 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81ae9a0 con 0x559fcb086620 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93e1987740 10 client.0 _ll_register_callbacks cb 0x7f93d0003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93dca0e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:53252/0 (socket says 172.21.15.26:53252) 2024-04-23T18:25:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93d7fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93dca0e640 1 -- 172.21.15.26:0/20677559 learned_addr learned my addr 172.21.15.26:0/20677559 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.904+0000 7f93dd20f640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb096f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 -- 172.21.15.26:0/20677559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 msgr2=0x559fcb099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 -- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 msgr2=0x559fcb096f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb096f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 -- 172.21.15.26:0/20677559 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fcb0977b0 con 0x559fcb082b30 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93d7fff640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dd20f640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb096f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93dca0e640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f93c80bd1f0 tx=0x7f93c806c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d81960ed7444ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8099120 con 0x559fcb082b30 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f93c80bbbc0 con 0x559fcb082b30 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93cc015050 con 0x559fcb082b30 2024-04-23T18:25:54.902 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f93c80992c0 con 0x559fcb082b30 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93e1987740 10 client.16701 fetch_fsmap learned FSMap version 44 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.905+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559fcb107a10 con 0x559fcb082b30 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93cc015050 con 0x559fcb082b30 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f93c8018b20 con 0x559fcb082b30 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 10 client.16701 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 20 client.16701 populate_metadata read hostname 'smithi026' 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb091800 0x559fcb10a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 4 client.16701 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559fcaff3200 con 0x559fcb091800 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93d7fff640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb091800 0x559fcb10a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93b7fff640 10 client.16701 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:54.903 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.906+0000 7f93d7fff640 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb091800 0x559fcb10a150 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd751ba0b1002f11 server_cookie=cf32635ead6d79d6 in_seq=0 out_seq=0 2024-04-23T18:25:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93b7fff640 1 -- 172.21.15.26:0/20677559 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559fcaff3200 con 0x559fcb091800 2024-04-23T18:25:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93b7fff640 10 client.16701 handle_command_reply: tid=0 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 client.16701 shutdown 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 msgr2=0x559fcb099830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f93c80bd1f0 tx=0x7f93c806c450 comp rx=0 tx=0).stop 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 shutdown_connections 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x559fcb091800 0x559fcb10a150 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559fcb086f40 0x559fcb096f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93b7fff640 0 client.16701 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559fcb086620 0x559fcb099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 --2- 172.21.15.26:0/20677559 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x559fcb082b30 0x559fcb099830 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.940+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 >> 172.21.15.26:0/20677559 conn(0x559fcb08ea80 msgr2=0x559fcb081cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.941+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 shutdown_connections 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.941+0000 7f93e1987740 1 -- 172.21.15.26:0/20677559 wait complete. 2024-04-23T18:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.941+0000 7f93e1987740 20 client.16701 trim_cache size 0 max 16384 2024-04-23T18:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 msgr2=0x7f93c00a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 0x7f93c00a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009430 tx=0x7f93d0005c50 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 msgr2=0x7f93d8100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9f0 tx=0x7f93c800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 shutdown_connections 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f93c00a3e70 0x7f93c00a6330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93d810c550 0x7f93d8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f93d8103aa0 0x7f93d8101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 --2- 172.21.15.26:0/1279762469 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f93d8103180 0x7f93d8100dd0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 >> 172.21.15.26:0/1279762469 conn(0x7f93d8077fd0 msgr2=0x7f93d80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 shutdown_connections 2024-04-23T18:25:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:25:54.942+0000 7f93dec72640 1 -- 172.21.15.26:0/1279762469 wait complete. 2024-04-23T18:25:55.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:54 smithi110 ceph-mon[32585]: pgmap v270: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-23T18:25:55.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:54 smithi026 ceph-mon[29364]: pgmap v270: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-23T18:25:55.240 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:54 smithi131 ceph-mon[33041]: pgmap v270: 97 pgs: 97 active+clean; 340 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-23T18:25:55.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 529697 5.80 MB/sec execute 350 sec latency 17.652 ms 2024-04-23T18:25:56.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 531541 5.80 MB/sec execute 351 sec latency 19.770 ms 2024-04-23T18:25:57.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:56 smithi110 ceph-mon[32585]: pgmap v271: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2024-04-23T18:25:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:56 smithi026 ceph-mon[29364]: pgmap v271: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2024-04-23T18:25:57.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:56 smithi131 ceph-mon[33041]: pgmap v271: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2024-04-23T18:25:57.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 533319 5.82 MB/sec execute 352 sec latency 15.028 ms 2024-04-23T18:25:58.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 534978 5.82 MB/sec execute 353 sec latency 44.873 ms 2024-04-23T18:25:59.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:25:58 smithi110 ceph-mon[32585]: pgmap v272: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:25:59.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:25:58 smithi026 ceph-mon[29364]: pgmap v272: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:25:59.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:25:58 smithi131 ceph-mon[33041]: pgmap v272: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:25:59.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 536833 5.84 MB/sec execute 354 sec latency 14.085 ms 2024-04-23T18:26:00.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 538695 5.84 MB/sec execute 355 sec latency 18.040 ms 2024-04-23T18:26:01.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:00 smithi110 ceph-mon[32585]: pgmap v273: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:26:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:00 smithi026 ceph-mon[29364]: pgmap v273: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:26:01.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:00 smithi131 ceph-mon[33041]: pgmap v273: 97 pgs: 97 active+clean; 349 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-23T18:26:01.400 INFO:tasks.quiescer.fs.[cephfs]:Total duration (47.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:26:01.400 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.65 2024-04-23T18:26:01.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:01.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 -- 172.21.15.26:0/2572117913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4101090 msgr2=0x7fa7a4101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2572117913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4101090 0x7fa7a4101470 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa794009900 tx=0x7fa79402a410 comp rx=0 tx=0).stop 2024-04-23T18:26:01.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 -- 172.21.15.26:0/2572117913 shutdown_connections 2024-04-23T18:26:01.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2572117913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a4101a40 0x7fa7a4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2572117913 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4106160 0x7fa7a410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2572117913 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4101090 0x7fa7a4101470 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 -- 172.21.15.26:0/2572117913 >> 172.21.15.26:0/2572117913 conn(0x7fa7a40771d0 msgr2=0x7fa7a40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 -- 172.21.15.26:0/2572117913 shutdown_connections 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.521+0000 7fa7a909e640 1 -- 172.21.15.26:0/2572117913 wait complete. 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 Processor -- start 2024-04-23T18:26:01.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 -- start start 2024-04-23T18:26:01.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4106160 0x7fa7a4103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 0x7fa7a41a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a4111120 con 0x7fa7a4106160 2024-04-23T18:26:01.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a4110fa0 con 0x7fa7a41035a0 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.522+0000 7fa7a909e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a41112a0 con 0x7fa7a4101090 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4106160 0x7fa7a4103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 0x7fa7a41a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:51370/0 (socket says 172.21.15.26:51370) 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 -- 172.21.15.26:0/2237466684 learned_addr learned my addr 172.21.15.26:0/2237466684 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 -- 172.21.15.26:0/2237466684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 msgr2=0x7fa7a41a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 0x7fa7a41a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 -- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4106160 msgr2=0x7fa7a4103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4106160 0x7fa7a4103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 -- 172.21.15.26:0/2237466684 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a41a7030 con 0x7fa7a4101090 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a359e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 0x7fa7a41a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:01.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a2d9d640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa794005ab0 tx=0x7fa794003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79403f070 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a41a72c0 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa794031b60 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794035030 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.523+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a41a7800 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.524+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a4102560 con 0x7fa7a4101090 2024-04-23T18:26:01.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.526+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa794042070 con 0x7fa7a4101090 2024-04-23T18:26:01.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.526+0000 7fa78bfff640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 0x7fa78c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.526+0000 7fa7a259c640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 0x7fa78c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.526+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7fa7940f1480 con 0x7fa7a4101090 2024-04-23T18:26:01.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.527+0000 7fa7a259c640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 0x7fa78c0a6210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa7a41045b0 tx=0x7fa798005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.530+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7940bd4b0 con 0x7fa7a4101090 2024-04-23T18:26:01.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540064 5.86 MB/sec execute 356 sec latency 105.816 ms 2024-04-23T18:26:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.780+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7a410a3a0 con 0x7fa7a4101090 2024-04-23T18:26:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.781+0000 7fa78bfff640 1 -- 172.21.15.26:0/2237466684 <== mon.1 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11885 (secure 0 0 0) 0x7fa7940c17b0 con 0x7fa7a4101090 2024-04-23T18:26:01.774 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:26:01.774 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:25:35.675891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24460,24464]},"id":1}]} 2024-04-23T18:26:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 44 2024-04-23T18:26:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 msgr2=0x7fa78c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 0x7fa78c0a6210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa7a41045b0 tx=0x7fa798005f90 comp rx=0 tx=0).stop 2024-04-23T18:26:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 msgr2=0x7fa7a4102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa794005ab0 tx=0x7fa794003c10 comp rx=0 tx=0).stop 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 shutdown_connections 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa78c0a3d50 0x7fa78c0a6210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7a41035a0 0x7fa7a41a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa7a4106160 0x7fa7a4103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 --2- 172.21.15.26:0/2237466684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa7a4101090 0x7fa7a4102b20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.783+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 >> 172.21.15.26:0/2237466684 conn(0x7fa7a40771d0 msgr2=0x7fa7a40fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.784+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 shutdown_connections 2024-04-23T18:26:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.784+0000 7fa7a909e640 1 -- 172.21.15.26:0/2237466684 wait complete. 2024-04-23T18:26:01.788 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 83.65 --await-for 42.0' 2024-04-23T18:26:01.789 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 83.65 --await-for 42.0 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 -- 172.21.15.26:0/140206291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50101a50 msgr2=0x7f8c50105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 --2- 172.21.15.26:0/140206291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50101a50 0x7f8c50105a90 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009900 tx=0x7f8c3c02a410 comp rx=0 tx=0).stop 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 -- 172.21.15.26:0/140206291 shutdown_connections 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 --2- 172.21.15.26:0/140206291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c50106170 0x7f8c5010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 --2- 172.21.15.26:0/140206291 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50101a50 0x7f8c50105a90 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 --2- 172.21.15.26:0/140206291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501010a0 0x7f8c50101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 -- 172.21.15.26:0/140206291 >> 172.21.15.26:0/140206291 conn(0x7f8c50077f50 msgr2=0x7f8c50078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 -- 172.21.15.26:0/140206291 shutdown_connections 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.916+0000 7f8c56d76640 1 -- 172.21.15.26:0/140206291 wait complete. 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 Processor -- start 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 -- start start 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c501010a0 0x7f8c50102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501110b0 con 0x7f8c50106170 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50110f30 con 0x7f8c501035b0 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c56d76640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50111230 con 0x7f8c501010a0 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57566/0 (socket says 172.21.15.26:57566) 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:36670/0 (socket says 172.21.15.26:36670) 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c4ffff640 1 -- 172.21.15.26:0/372835260 learned_addr learned my addr 172.21.15.26:0/372835260 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c54b12640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c501010a0 0x7f8c50102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 -- 172.21.15.26:0/372835260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c501010a0 msgr2=0x7f8c50102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c501010a0 0x7f8c50102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 -- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 msgr2=0x7f8c50103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.917+0000 7f8c55313640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c55313640 1 -- 172.21.15.26:0/372835260 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c501a7010 con 0x7f8c501035b0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c55313640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009fd0 tx=0x7f8c4800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c4dffb640 1 -- 172.21.15.26:0/372835260 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800ce20 con 0x7f8c501035b0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c4dffb640 1 -- 172.21.15.26:0/372835260 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c48005020 con 0x7f8c501035b0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c501a7300 con 0x7f8c501035b0 2024-04-23T18:26:01.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c4dffb640 1 -- 172.21.15.26:0/372835260 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48010690 con 0x7f8c501035b0 2024-04-23T18:26:01.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.918+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c501a78a0 con 0x7f8c501035b0 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.920+0000 7f8c4dffb640 1 -- 172.21.15.26:0/372835260 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8c480054d0 con 0x7f8c501035b0 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 Processor -- start 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 -- start start 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 0x564d97560b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d97561090 0x564d97563480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564d974bb370 con 0x564d975560a0 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564d974582e0 con 0x564d975569c0 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c59b69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564d974b7330 con 0x564d97561090 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57570/0 (socket says 172.21.15.26:57570) 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 -- 172.21.15.26:0/1494207008 learned_addr learned my addr 172.21.15.26:0/1494207008 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4dffb640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 0x7f8c400a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c54b12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 0x564d97560b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d97561090 0x564d97563480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c55313640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 0x7f8c400a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4dffb640 1 -- 172.21.15.26:0/372835260 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f8c480d0d80 con 0x7f8c501035b0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 -- 172.21.15.26:0/1494207008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d97561090 msgr2=0x564d97563480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d97561090 0x564d97563480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 -- 172.21.15.26:0/1494207008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 msgr2=0x564d97560b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 0x564d97560b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c4ffff640 1 -- 172.21.15.26:0/1494207008 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d97299d80 con 0x564d975560a0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.921+0000 7f8c54b12640 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 0x564d97560b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.922+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c032fd0 tx=0x7f8c3c005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc99515778f41870 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.922+0000 7f8c55313640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 0x7f8c400a6290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ce560 tx=0x7f8c48092190 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.922+0000 7f8c3affd640 1 -- 172.21.15.26:0/1494207008 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c044070 con 0x564d975560a0 2024-04-23T18:26:01.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.922+0000 7f8c3affd640 1 -- 172.21.15.26:0/1494207008 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c3c005020 con 0x564d975560a0 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.922+0000 7f8c3affd640 1 -- 172.21.15.26:0/1494207008 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c035040 con 0x564d975560a0 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- 172.21.15.26:0/1494207008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 msgr2=0x564d97556480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c032fd0 tx=0x7f8c3c005bc0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- 172.21.15.26:0/1494207008 shutdown_connections 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975569c0 0x564d97560b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d97561090 0x564d97563480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1494207008 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97556480 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- 172.21.15.26:0/1494207008 >> 172.21.15.26:0/1494207008 conn(0x564d975523c0 msgr2=0x564d975527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- 172.21.15.26:0/1494207008 shutdown_connections 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- 172.21.15.26:0/1494207008 wait complete. 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- start start 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 Processor -- start 2024-04-23T18:26:01.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975560a0 0x564d9755e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x564d97563e20 con 0x564d97561090 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564d97455fb0 con 0x564d975560a0 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.923+0000 7f8c59b69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x564d97563fa0 con 0x564d975569c0 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c59b69740 10 client.0 _ll_register_callbacks cb 0x564d974559a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:51408/0 (socket says 172.21.15.26:51408) 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57578/0 (socket says 172.21.15.26:57578) 2024-04-23T18:26:01.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 -- 172.21.15.26:0/1680522935 learned_addr learned my addr 172.21.15.26:0/1680522935 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975560a0 0x564d9755e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 -- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975560a0 msgr2=0x564d9755e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975560a0 0x564d9755e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 -- 172.21.15.26:0/1680522935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 msgr2=0x564d9755ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 -- 172.21.15.26:0/1680522935 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d972fa510 con 0x564d975569c0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c54b12640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8c44012740 tx=0x7f8c44012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa2c7eaa95b1af89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44004030 con 0x564d975569c0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mon.1 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f8c440041d0 con 0x564d975569c0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.924+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c005020 con 0x564d975569c0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8c440043d0 con 0x564d975569c0 2024-04-23T18:26:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 10 client.26044 fetch_fsmap learned FSMap version 44 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564d9718d670 con 0x564d975569c0 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mon.1 v2:172.21.15.131:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c3c005020 con 0x564d975569c0 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mon.1 v2:172.21.15.131:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f8c44003b50 con 0x564d975569c0 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 10 client.26044 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 10 client.26044 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 20 client.26044 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d97571460 0x564d97573880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 4 client.26044 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d974c8d90 con 0x564d97571460 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.925+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d97571460 0x564d97573880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.926+0000 7f8c2ffff640 10 client.26044 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.926+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d97571460 0x564d97573880 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3907eacf09728aa2 server_cookie=b52cb5b2bfe9e5c6 in_seq=0 out_seq=0 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c2ffff640 1 -- 172.21.15.26:0/1680522935 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d974c8d90 con 0x564d97571460 2024-04-23T18:26:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c2ffff640 10 client.26044 handle_command_reply: tid=0 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 client.26044 shutdown 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 msgr2=0x564d9755e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8c44012740 tx=0x7f8c44012c00 comp rx=0 tx=0).stop 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 shutdown_connections 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d97571460 0x564d97573880 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d97561090 0x564d9755ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755e590 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c2ffff640 0 client.26044 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 --2- 172.21.15.26:0/1680522935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d975560a0 0x564d9755e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 >> 172.21.15.26:0/1680522935 conn(0x564d975523c0 msgr2=0x564d97551600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 shutdown_connections 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 1 -- 172.21.15.26:0/1680522935 wait complete. 2024-04-23T18:26:01.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.927+0000 7f8c59b69740 20 client.26044 trim_cache size 0 max 16384 2024-04-23T18:26:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 Processor -- start 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 -- start start 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97551d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d9755e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501b2680 con 0x564d975560a0 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501aff50 con 0x564d97561090 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c59b69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501addc0 con 0x564d975569c0 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97551d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:51418/0 (socket says 172.21.15.26:51418) 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 -- 172.21.15.26:0/527437530 learned_addr learned my addr 172.21.15.26:0/527437530 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d9755e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 -- 172.21.15.26:0/527437530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 msgr2=0x564d9755e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d9755e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 -- 172.21.15.26:0/527437530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 msgr2=0x564d97551d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97551d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.946+0000 7f8c54b12640 1 -- 172.21.15.26:0/527437530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d975567f0 con 0x564d975569c0 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c54b12640 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c44000c00 tx=0x7f8c44005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=858bfae33bdb0325 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c3a7fc640 1 -- 172.21.15.26:0/527437530 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44029030 con 0x564d975569c0 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c3a7fc640 1 -- 172.21.15.26:0/527437530 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c44012e50 con 0x564d975569c0 2024-04-23T18:26:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c3a7fc640 1 -- 172.21.15.26:0/527437530 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44003770 con 0x564d975569c0 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 -- 172.21.15.26:0/527437530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 msgr2=0x564d9755d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c44000c00 tx=0x7f8c44005810 comp rx=0 tx=0).stop 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 -- 172.21.15.26:0/527437530 shutdown_connections 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d9755e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d9755d930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 --2- 172.21.15.26:0/527437530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97551d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.947+0000 7f8c59b69740 1 -- 172.21.15.26:0/527437530 >> 172.21.15.26:0/527437530 conn(0x564d975523c0 msgr2=0x564d9755e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- 172.21.15.26:0/527437530 shutdown_connections 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- 172.21.15.26:0/527437530 wait complete. 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- start start 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 Processor -- start 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d97569650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d97564600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501ae410 con 0x564d975560a0 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501af880 con 0x564d97561090 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501aeae0 con 0x564d975569c0 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.948+0000 7f8c59b69740 10 client.0 _ll_register_callbacks cb 0x7f8c44009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:57588/0 (socket says 172.21.15.26:57588) 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 -- 172.21.15.26:0/4167714788 learned_addr learned my addr 172.21.15.26:0/4167714788 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c54b12640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d97569650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d97564600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 -- 172.21.15.26:0/4167714788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 msgr2=0x564d97569650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d97569650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 -- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 msgr2=0x564d97564600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d97564600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 -- 172.21.15.26:0/4167714788 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d97564e30 con 0x564d975560a0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c4ffff640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006010 tx=0x7f8c3c03c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b9f87c9bece5d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c044040 con 0x564d975560a0 2024-04-23T18:26:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f8c3c056ae0 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.949+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c44012e50 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8c3c035040 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 10 client.16725 fetch_fsmap learned FSMap version 44 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564d975d7500 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c44012e50 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f8c3c056ce0 con 0x564d975560a0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 10 client.16725 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 10 client.16725 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 20 client.16725 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d975d7780 0x564d975d9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 4 client.16725 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.65, "await_for": 42.0}] 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.65, "await_for": 42.0}) -- 0x564d974c2c00 con 0x564d975d7780 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.950+0000 7f8c54b12640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d975d7780 0x564d975d9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.951+0000 7f8c2ffff640 10 client.16725 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:01.951+0000 7f8c54b12640 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d975d7780 0x564d975d9b70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afe8aa01e842ec35 server_cookie=abb37ef6fbddd5b6 in_seq=0 out_seq=0 2024-04-23T18:26:02.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.038+0000 7f8c2ffff640 1 -- 172.21.15.26:0/4167714788 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x564d974c2c00 con 0x564d975d7780 2024-04-23T18:26:02.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.038+0000 7f8c2ffff640 10 client.16725 handle_command_reply: tid=0 2024-04-23T18:26:02.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.038+0000 7f8c59b69740 1 client.16725 shutdown 2024-04-23T18:26:02.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 msgr2=0x564d97569110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006010 tx=0x7f8c3c03c040 comp rx=0 tx=0).stop 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 shutdown_connections 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x564d975d7780 0x564d975d9b70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564d97561090 0x564d97564600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564d975569c0 0x564d97569650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c2ffff640 0 client.16725 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 --2- 172.21.15.26:0/4167714788 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x564d975560a0 0x564d97569110 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 >> 172.21.15.26:0/4167714788 conn(0x564d975523c0 msgr2=0x564d9755e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 shutdown_connections 2024-04-23T18:26:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 1 -- 172.21.15.26:0/4167714788 wait complete. 2024-04-23T18:26:02.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.039+0000 7f8c59b69740 20 client.16725 trim_cache size 0 max 16384 2024-04-23T18:26:02.036 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:26:02.036 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 44, 2024-04-23T18:26:02.036 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:26:02.036 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 38, 2024-04-23T18:26:02.036 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "73106a2a": { 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 38, 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:26:02.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 83.7, 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:26:02.038 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:02.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:02.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:02.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:02.040 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:26:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 msgr2=0x7f8c400a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 0x7f8c400a6290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ce560 tx=0x7f8c48092190 comp rx=0 tx=0).stop 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 msgr2=0x7f8c501a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009fd0 tx=0x7f8c4800ef10 comp rx=0 tx=0).stop 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 shutdown_connections 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8c400a3dd0 0x7f8c400a6290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c501035b0 0x7f8c501a6910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8c50106170 0x7f8c50103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 --2- 172.21.15.26:0/372835260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c501010a0 0x7f8c50102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 >> 172.21.15.26:0/372835260 conn(0x7f8c50077f50 msgr2=0x7f8c501001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 shutdown_connections 2024-04-23T18:26:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:02.040+0000 7f8c56d76640 1 -- 172.21.15.26:0/372835260 wait complete. 2024-04-23T18:26:02.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:01 smithi026 ceph-mon[29364]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:02.043 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:01 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2237466684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:02.044 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '73106a2a', quiesce took 0.6 seconds. Will release after: 23.65 2024-04-23T18:26:02.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:01 smithi110 ceph-mon[32585]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:02.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:01 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2237466684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:02.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:01 smithi131 ceph-mon[33041]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:02.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:01 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2237466684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:02.687 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.84 MB/sec execute 357 sec latency 657.631 ms 2024-04-23T18:26:03.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:02 smithi110 ceph-mon[32585]: pgmap v275: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:02 smithi026 ceph-mon[29364]: pgmap v275: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:03.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:02 smithi131 ceph-mon[33041]: pgmap v275: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:03.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.83 MB/sec execute 358 sec latency 1657.690 ms 2024-04-23T18:26:04.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.81 MB/sec execute 359 sec latency 2657.799 ms 2024-04-23T18:26:05.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:04 smithi110 ceph-mon[32585]: pgmap v276: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:05.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:04 smithi026 ceph-mon[29364]: pgmap v276: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:05.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:04 smithi131 ceph-mon[33041]: pgmap v276: 97 pgs: 97 active+clean; 357 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2024-04-23T18:26:05.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.80 MB/sec execute 360 sec latency 3657.898 ms 2024-04-23T18:26:06.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.78 MB/sec execute 361 sec latency 4658.021 ms 2024-04-23T18:26:07.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:06 smithi110 ceph-mon[32585]: pgmap v277: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-23T18:26:07.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:06 smithi110 ceph-mon[32585]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:07.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:06 smithi026 ceph-mon[29364]: pgmap v277: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-23T18:26:07.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:06 smithi026 ceph-mon[29364]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:07.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:06 smithi131 ceph-mon[33041]: pgmap v277: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-23T18:26:07.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:06 smithi131 ceph-mon[33041]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:07.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.76 MB/sec execute 362 sec latency 5658.137 ms 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.041+0000 7f73a8a4e640 1 -- 172.21.15.26:0/1261648721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a410c5d0 msgr2=0x7f73a410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.041+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/1261648721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a410c5d0 0x7f73a410ea00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f739c009ca0 tx=0x7f739c02a950 comp rx=0 tx=0).stop 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 -- 172.21.15.26:0/1261648721 shutdown_connections 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/1261648721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a4103b20 0x7f73a410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/1261648721 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a410c5d0 0x7f73a410ea00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/1261648721 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103200 0x7f73a41035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 -- 172.21.15.26:0/1261648721 >> 172.21.15.26:0/1261648721 conn(0x7f73a40771d0 msgr2=0x7f73a40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 -- 172.21.15.26:0/1261648721 shutdown_connections 2024-04-23T18:26:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 -- 172.21.15.26:0/1261648721 wait complete. 2024-04-23T18:26:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.042+0000 7f73a8a4e640 1 Processor -- start 2024-04-23T18:26:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 -- start start 2024-04-23T18:26:08.037 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 0x7f73a4101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 0x7f73a41018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41121f0 con 0x7f73a4103200 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73a4112070 con 0x7f73a410c5d0 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a8a4e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f73a4112370 con 0x7f73a4103b20 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37744/0 (socket says 172.21.15.26:37744) 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 -- 172.21.15.26:0/2279211622 learned_addr learned my addr 172.21.15.26:0/2279211622 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 0x7f73a4101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 0x7f73a41018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 -- 172.21.15.26:0/2279211622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 msgr2=0x7f73a4101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 0x7f73a4101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 -- 172.21.15.26:0/2279211622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 msgr2=0x7f73a41018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 0x7f73a41018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a259c640 1 -- 172.21.15.26:0/2279211622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a41a6ab0 con 0x7f73a4103200 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 0x7f73a41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.043+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 0x7f73a4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73a259c640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f739400d770 tx=0x7f739400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73937fe640 1 -- 172.21.15.26:0/2279211622 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394017070 con 0x7f73a4103200 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a41a6da0 con 0x7f73a4103200 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73937fe640 1 -- 172.21.15.26:0/2279211622 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73940054d0 con 0x7f73a4103200 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73937fe640 1 -- 172.21.15.26:0/2279211622 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394004c30 con 0x7f73a4103200 2024-04-23T18:26:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.044+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a41a7210 con 0x7f73a4103200 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73937fe640 1 -- 172.21.15.26:0/2279211622 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7394004df0 con 0x7f73a4103200 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 Processor -- start 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 -- start start 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df85da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5600df7c2970 con 0x5600df85df40 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5600df75f4a0 con 0x5600df868800 2024-04-23T18:26:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73ab765740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5600df7be930 con 0x5600df85d620 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37758/0 (socket says 172.21.15.26:37758) 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73a259c640 1 -- 172.21.15.26:0/2586973775 learned_addr learned my addr 172.21.15.26:0/2586973775 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df85da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.047+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73937fe640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 0x7f738c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 0x7f738c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73937fe640 1 -- 172.21.15.26:0/2279211622 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f73940d0400 con 0x7f73a4103200 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 -- 172.21.15.26:0/2586973775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 msgr2=0x5600df85da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df85da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 -- 172.21.15.26:0/2586973775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 msgr2=0x5600df86abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 -- 172.21.15.26:0/2586973775 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600df4d5210 con 0x5600df85df40 2024-04-23T18:26:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a259c640 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f73940a4a80 tx=0x7f7394009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c21792de2e82a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f7390ff9640 1 -- 172.21.15.26:0/2586973775 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394020070 con 0x5600df85df40 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 0x7f738c0a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f739c030040 tx=0x7f739c02a950 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f7390ff9640 1 -- 172.21.15.26:0/2586973775 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f739409ca40 con 0x5600df85df40 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.048+0000 7f7390ff9640 1 -- 172.21.15.26:0/2586973775 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73940989b0 con 0x5600df85df40 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 -- 172.21.15.26:0/2586973775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 msgr2=0x5600df8682c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f73940a4a80 tx=0x7f7394009f90 comp rx=0 tx=0).stop 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 -- 172.21.15.26:0/2586973775 shutdown_connections 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df8682c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 --2- 172.21.15.26:0/2586973775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df85da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 -- 172.21.15.26:0/2586973775 >> 172.21.15.26:0/2586973775 conn(0x5600df859b30 msgr2=0x5600df859f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 -- 172.21.15.26:0/2586973775 shutdown_connections 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.049+0000 7f73ab765740 1 -- 172.21.15.26:0/2586973775 wait complete. 2024-04-23T18:26:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 -- start start 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 Processor -- start 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df868110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df865400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5600df86b3d0 con 0x5600df868800 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5600df75d0b0 con 0x5600df85df40 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5600df86b550 con 0x5600df85d620 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73ab765740 10 client.0 _ll_register_callbacks cb 0x5600df74ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37760/0 (socket says 172.21.15.26:37760) 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 -- 172.21.15.26:0/505398383 learned_addr learned my addr 172.21.15.26:0/505398383 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df865400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a259c640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df868110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 -- 172.21.15.26:0/505398383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 msgr2=0x5600df868110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df868110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 -- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 msgr2=0x5600df865400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df865400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a2d9d640 1 -- 172.21.15.26:0/505398383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600df5a1620 con 0x5600df868800 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.050+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df865400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f739c035000 tx=0x7f739c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683d4c50ffc73595 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c04d030 con 0x5600df868800 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f739c0043d0 con 0x5600df868800 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f739409ca40 con 0x5600df868800 2024-04-23T18:26:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f739c034040 con 0x5600df868800 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 10 client.16737 fetch_fsmap learned FSMap version 44 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5600df877340 con 0x5600df868800 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f739409ca40 con 0x5600df868800 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f739c048040 con 0x5600df868800 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 10 client.16737 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 20 client.16737 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df878870 0x5600df87ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 4 client.16737 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.051+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600df7d0390 con 0x5600df878870 2024-04-23T18:26:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.052+0000 7f73a259c640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df878870 0x5600df87ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.052+0000 7f73a259c640 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df878870 0x5600df87ac60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f275f14c7095b074 server_cookie=1670f4cac9eafc5a in_seq=0 out_seq=0 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.052+0000 7f7389ffb640 10 client.16737 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f7389ffb640 1 -- 172.21.15.26:0/505398383 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600df7d0390 con 0x5600df878870 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f7389ffb640 10 client.16737 handle_command_reply: tid=0 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 client.16737 shutdown 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 msgr2=0x5600df865940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f739c035000 tx=0x7f739c0095c0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 shutdown_connections 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df878870 0x5600df87ac60 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df865400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df868800 0x5600df865940 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f7389ffb640 0 client.16737 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 --2- 172.21.15.26:0/505398383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df868110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 >> 172.21.15.26:0/505398383 conn(0x5600df859b30 msgr2=0x5600df85f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 shutdown_connections 2024-04-23T18:26:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 1 -- 172.21.15.26:0/505398383 wait complete. 2024-04-23T18:26:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.053+0000 7f73ab765740 20 client.16737 trim_cache size 0 max 16384 2024-04-23T18:26:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 Processor -- start 2024-04-23T18:26:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 -- start start 2024-04-23T18:26:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df860970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df868800 0x5600df867c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41b24f0 con 0x5600df85d620 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41afdc0 con 0x5600df85df40 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73ab765740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41adc30 con 0x5600df868800 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73a259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37770/0 (socket says 172.21.15.26:37770) 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73a259c640 1 -- 172.21.15.26:0/1100270033 learned_addr learned my addr 172.21.15.26:0/1100270033 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.069+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df860970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 -- 172.21.15.26:0/1100270033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df868800 msgr2=0x5600df867c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df868800 0x5600df867c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 -- 172.21.15.26:0/1100270033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 msgr2=0x5600df860970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df860970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 -- 172.21.15.26:0/1100270033 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600df85dd70 con 0x5600df85d620 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df860970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73a259c640 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7394009610 tx=0x7f7394005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5746969259a8e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f738bfff640 1 -- 172.21.15.26:0/1100270033 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394020040 con 0x5600df85d620 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f738bfff640 1 -- 172.21.15.26:0/1100270033 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f739409ca40 con 0x5600df85d620 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f738bfff640 1 -- 172.21.15.26:0/1100270033 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73940cdd70 con 0x5600df85d620 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 -- 172.21.15.26:0/1100270033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 msgr2=0x5600df874290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7394009610 tx=0x7f7394005780 comp rx=0 tx=0).stop 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 -- 172.21.15.26:0/1100270033 shutdown_connections 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df85df40 0x5600df860970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df868800 0x5600df867c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 --2- 172.21.15.26:0/1100270033 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85d620 0x5600df874290 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 -- 172.21.15.26:0/1100270033 >> 172.21.15.26:0/1100270033 conn(0x5600df859b30 msgr2=0x5600df860260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 -- 172.21.15.26:0/1100270033 shutdown_connections 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.070+0000 7f73ab765740 1 -- 172.21.15.26:0/1100270033 wait complete. 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 -- start start 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 Processor -- start 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df8707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41ae280 con 0x5600df85df40 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41af6f0 con 0x5600df868800 2024-04-23T18:26:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41ae950 con 0x5600df85d620 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73ab765740 10 client.0 _ll_register_callbacks cb 0x7f739c02fe00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73a1d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73a1d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:37776/0 (socket says 172.21.15.26:37776) 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73a1d9b640 1 -- 172.21.15.26:0/591189796 learned_addr learned my addr 172.21.15.26:0/591189796 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73a259c640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df8707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.071+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 -- 172.21.15.26:0/591189796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 msgr2=0x5600df8707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df8707c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 -- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 msgr2=0x5600df86bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 -- 172.21.15.26:0/591189796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600df86c730 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a2d9d640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73a1d9b640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x5600df85acc0 tx=0x7f7398012e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a179f72a1a682e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f7389ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398004030 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f73980041d0 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f739409ca40 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f73980043d0 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73ab765740 10 client.16743 fetch_fsmap learned FSMap version 44 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5600df8dea40 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.072+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f739409ca40 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 44) ==== 8342+0+0 (secure 0 0 0) 0x7f7398003ae0 con 0x5600df85df40 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 10 client.16743 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:26:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 20 client.16743 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df8df220 0x5600df8e1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 4 client.16743 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600df865cd0 con 0x5600df8df220 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73a259c640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df8df220 0x5600df8e1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f73a259c640 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df8df220 0x5600df8e1640 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=108a13f3701fb181 server_cookie=48a5a5e1c3c5a8cc in_seq=0 out_seq=0 2024-04-23T18:26:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.073+0000 7f7389ffb640 10 client.16743 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.094+0000 7f7389ffb640 1 -- 172.21.15.26:0/591189796 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600df865cd0 con 0x5600df8df220 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.094+0000 7f7389ffb640 10 client.16743 handle_command_reply: tid=0 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.094+0000 7f73ab765740 1 client.16743 shutdown 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 msgr2=0x5600df870d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x5600df85acc0 tx=0x7f7398012e10 comp rx=0 tx=0).stop 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 shutdown_connections 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5600df8df220 0x5600df8e1640 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5600df868800 0x5600df86bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5600df85df40 0x5600df870d00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 --2- 172.21.15.26:0/591189796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5600df85d620 0x5600df8707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f7389ffb640 0 client.16743 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 >> 172.21.15.26:0/591189796 conn(0x5600df859b30 msgr2=0x5600df858420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 shutdown_connections 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 1 -- 172.21.15.26:0/591189796 wait complete. 2024-04-23T18:26:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.095+0000 7f73ab765740 20 client.16743 trim_cache size 0 max 16384 2024-04-23T18:26:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:26:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:26:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:26:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 msgr2=0x7f738c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 0x7f738c0a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f739c030040 tx=0x7f739c02a950 comp rx=0 tx=0).stop 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 msgr2=0x7f73a4100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f739400d770 tx=0x7f739400dc30 comp rx=0 tx=0).stop 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 shutdown_connections 2024-04-23T18:26:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f738c0a3dc0 0x7f738c0a6280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73a410c5d0 0x7f73a41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f73a4103b20 0x7f73a4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 --2- 172.21.15.26:0/2279211622 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f73a4103200 0x7f73a4100e50 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 >> 172.21.15.26:0/2279211622 conn(0x7f73a40771d0 msgr2=0x7f73a40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 shutdown_connections 2024-04-23T18:26:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:08.096+0000 7f73a8a4e640 1 -- 172.21.15.26:0/2279211622 wait complete. 2024-04-23T18:26:08.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.75 MB/sec execute 363 sec latency 6658.286 ms 2024-04-23T18:26:09.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:08 smithi110 ceph-mon[32585]: pgmap v279: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 305 op/s 2024-04-23T18:26:09.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:08 smithi026 ceph-mon[29364]: pgmap v279: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 305 op/s 2024-04-23T18:26:09.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:08 smithi131 ceph-mon[33041]: pgmap v279: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 305 op/s 2024-04-23T18:26:09.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.73 MB/sec execute 364 sec latency 7658.357 ms 2024-04-23T18:26:10.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.72 MB/sec execute 365 sec latency 8658.463 ms 2024-04-23T18:26:11.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:10 smithi110 ceph-mon[32585]: pgmap v280: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 92 op/s 2024-04-23T18:26:11.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:10 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:11.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:10 smithi026 ceph-mon[29364]: pgmap v280: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 92 op/s 2024-04-23T18:26:11.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:10 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:11.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:10 smithi131 ceph-mon[33041]: pgmap v280: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 92 op/s 2024-04-23T18:26:11.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:10 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:11.688 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.70 MB/sec execute 366 sec latency 9658.527 ms 2024-04-23T18:26:12.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.68 MB/sec execute 367 sec latency 10658.604 ms 2024-04-23T18:26:13.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:12 smithi110 ceph-mon[32585]: pgmap v281: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:13.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:12 smithi026 ceph-mon[29364]: pgmap v281: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:13.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:12 smithi131 ceph-mon[33041]: pgmap v281: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:13.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.67 MB/sec execute 368 sec latency 11658.699 ms 2024-04-23T18:26:14.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.65 MB/sec execute 369 sec latency 12658.779 ms 2024-04-23T18:26:15.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:14 smithi110 ceph-mon[32585]: pgmap v282: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:15.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:14 smithi026 ceph-mon[29364]: pgmap v282: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:15.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:14 smithi131 ceph-mon[33041]: pgmap v282: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 117 op/s 2024-04-23T18:26:15.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.64 MB/sec execute 370 sec latency 13658.904 ms 2024-04-23T18:26:16.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.62 MB/sec execute 371 sec latency 14659.015 ms 2024-04-23T18:26:17.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:16 smithi110 ceph-mon[32585]: pgmap v283: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.9 MiB/s wr, 82 op/s 2024-04-23T18:26:17.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:16 smithi026 ceph-mon[29364]: pgmap v283: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.9 MiB/s wr, 82 op/s 2024-04-23T18:26:17.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:16 smithi131 ceph-mon[33041]: pgmap v283: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 1.9 MiB/s wr, 82 op/s 2024-04-23T18:26:17.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.61 MB/sec execute 372 sec latency 15659.129 ms 2024-04-23T18:26:18.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.59 MB/sec execute 373 sec latency 16659.184 ms 2024-04-23T18:26:19.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:18 smithi110 ceph-mon[32585]: pgmap v284: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-23T18:26:19.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:18 smithi026 ceph-mon[29364]: pgmap v284: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-23T18:26:19.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:18 smithi131 ceph-mon[33041]: pgmap v284: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-23T18:26:19.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.58 MB/sec execute 374 sec latency 17659.318 ms 2024-04-23T18:26:20.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.56 MB/sec execute 375 sec latency 18659.431 ms 2024-04-23T18:26:21.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:20 smithi110 ceph-mon[32585]: pgmap v285: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/4033666258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 msgr2=0x7fc0c8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/4033666258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c8106200 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009900 tx=0x7fc0c402a400 comp rx=0 tx=0).stop 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/4033666258 shutdown_connections 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/4033666258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c81011a0 0x7fc0c8101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/4033666258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c8101b40 0x7fc0c810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/4033666258 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c8106200 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/4033666258 >> 172.21.15.26:0/4033666258 conn(0x7fc0c8076c50 msgr2=0x7fc0c8077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/4033666258 shutdown_connections 2024-04-23T18:26:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.199+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/4033666258 wait complete. 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 Processor -- start 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 -- start start 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 0x7fc0c81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c8101b40 0x7fc0c8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8111170 con 0x7fc0c8105e20 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8110ff0 con 0x7fc0c8101b40 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.200+0000 7fc0cfb7d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81112f0 con 0x7fc0c81011a0 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38802/0 (socket says 172.21.15.26:38802) 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 -- 172.21.15.26:0/3035766369 learned_addr learned my addr 172.21.15.26:0/3035766369 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0cd118640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c8101b40 0x7fc0c8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 0x7fc0c81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 -- 172.21.15.26:0/3035766369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 msgr2=0x7fc0c81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 0x7fc0c81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 -- 172.21.15.26:0/3035766369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c8101b40 msgr2=0x7fc0c8198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c8101b40 0x7fc0c8198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 -- 172.21.15.26:0/3035766369 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c819d6c0 con 0x7fc0c8105e20 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 0x7fc0c81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0ce11a640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000e960 tx=0x7fc0c000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0beffd640 1 -- 172.21.15.26:0/3035766369 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0004d20 con 0x7fc0c8105e20 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0beffd640 1 -- 172.21.15.26:0/3035766369 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0c0005020 con 0x7fc0c8105e20 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c819d9b0 con 0x7fc0c8105e20 2024-04-23T18:26:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0beffd640 1 -- 172.21.15.26:0/3035766369 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c00116d0 con 0x7fc0c8105e20 2024-04-23T18:26:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.201+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c81a5240 con 0x7fc0c8105e20 2024-04-23T18:26:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.203+0000 7fc0beffd640 1 -- 172.21.15.26:0/3035766369 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc0c00051d0 con 0x7fc0c8105e20 2024-04-23T18:26:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.204+0000 7fc0d2974740 1 Processor -- start 2024-04-23T18:26:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.204+0000 7fc0d2974740 1 -- start start 2024-04-23T18:26:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22520 0x55f7f4f22900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f31c10 0x55f7f4f34000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4e87970 con 0x55f7f4f22e40 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4e244a0 con 0x55f7f4f22520 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0d2974740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4e83930 con 0x55f7f4f31c10 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0beffd640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 0x7fc0b40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38812/0 (socket says 172.21.15.26:38812) 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 -- 172.21.15.26:0/947714062 learned_addr learned my addr 172.21.15.26:0/947714062 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd118640 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f31c10 0x55f7f4f34000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd118640 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22520 0x55f7f4f22900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0beffd640 1 -- 172.21.15.26:0/3035766369 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc0c00d08c0 con 0x7fc0c8105e20 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0ce11a640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 0x7fc0b40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 -- 172.21.15.26:0/947714062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f31c10 msgr2=0x55f7f4f34000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f31c10 0x55f7f4f34000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 -- 172.21.15.26:0/947714062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22520 msgr2=0x55f7f4f22900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22520 0x55f7f4f22900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 -- 172.21.15.26:0/947714062 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f4b9a210 con 0x55f7f4f22e40 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0cd919640 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4033000 tx=0x7fc0c4032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877d108c29a104b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.205+0000 7fc0ce11a640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 0x7fc0b40a6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0099230 tx=0x7fc0c0087040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0b3fff640 1 -- 172.21.15.26:0/947714062 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c403f070 con 0x55f7f4f22e40 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0b3fff640 1 -- 172.21.15.26:0/947714062 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0c4004030 con 0x55f7f4f22e40 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0b3fff640 1 -- 172.21.15.26:0/947714062 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4035070 con 0x55f7f4f22e40 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0d2974740 1 -- 172.21.15.26:0/947714062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 msgr2=0x55f7f4f316d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0d2974740 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4033000 tx=0x7fc0c4032c60 comp rx=0 tx=0).stop 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0d2974740 1 -- 172.21.15.26:0/947714062 shutdown_connections 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.206+0000 7fc0d2974740 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f31c10 0x55f7f4f34000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.206 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f316d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 --2- 172.21.15.26:0/947714062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22520 0x55f7f4f22900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 -- 172.21.15.26:0/947714062 >> 172.21.15.26:0/947714062 conn(0x55f7f4f1eb30 msgr2=0x55f7f4f1ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 -- 172.21.15.26:0/947714062 shutdown_connections 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 -- 172.21.15.26:0/947714062 wait complete. 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 -- start start 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 Processor -- start 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.207+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0d2974740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4f347e0 con 0x55f7f4f31c10 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0d2974740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4e220b0 con 0x55f7f4f22e40 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0d2974740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4f34960 con 0x55f7f4f22520 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0d2974740 10 client.0 _ll_register_callbacks cb 0x55f7f4e0fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:21.207 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38824/0 (socket says 172.21.15.26:38824) 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 -- 172.21.15.26:0/3506071494 learned_addr learned my addr 172.21.15.26:0/3506071494 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 -- 172.21.15.26:0/3506071494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 msgr2=0x55f7f4f314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f314f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 -- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 msgr2=0x55f7f4f2e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 -- 172.21.15.26:0/3506071494 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f4c66620 con 0x55f7f4f31c10 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0cd118640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8012e20 tx=0x7fc0b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e491d87933f23a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8004050 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc0b80041f0 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.208+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0c4004030 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc0b80043f0 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 10 client.16755 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f7f4f40760 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0c4004030 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7fc0b8022440 con 0x55f7f4f31c10 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 10 client.16755 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 20 client.16755 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4f41c20 0x55f7f4f44010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 4 client.16755 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7f4e95390 con 0x55f7f4f41c20 2024-04-23T18:26:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.209+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4f41c20 0x55f7f4f44010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.210+0000 7fc0cd919640 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4f41c20 0x55f7f4f44010 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d7401dae0af859 server_cookie=7c63732bd8b0968e in_seq=0 out_seq=0 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.210+0000 7fc0b17fa640 10 client.16755 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.210+0000 7fc0b17fa640 1 -- 172.21.15.26:0/3506071494 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7f4e95390 con 0x55f7f4f41c20 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.210+0000 7fc0b17fa640 10 client.16755 handle_command_reply: tid=0 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 client.16755 shutdown 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 msgr2=0x55f7f4f2ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8012e20 tx=0x7fc0b80046c0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 shutdown_connections 2024-04-23T18:26:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4f41c20 0x55f7f4f44010 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2ed20 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0b17fa640 0 client.16755 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 --2- 172.21.15.26:0/3506071494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 >> 172.21.15.26:0/3506071494 conn(0x55f7f4f1eb30 msgr2=0x55f7f4f249a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 shutdown_connections 2024-04-23T18:26:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 1 -- 172.21.15.26:0/3506071494 wait complete. 2024-04-23T18:26:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.211+0000 7fc0d2974740 20 client.16755 trim_cache size 0 max 16384 2024-04-23T18:26:21.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:20 smithi026 ceph-mon[29364]: pgmap v285: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-04-23T18:26:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.226+0000 7fc0d2974740 1 Processor -- start 2024-04-23T18:26:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.226+0000 7fc0d2974740 1 -- start start 2024-04-23T18:26:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22e40 0x55f7f4f29d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f31020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81ae130 con 0x55f7f4f22520 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81ab9d0 con 0x55f7f4f31c10 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0d2974740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81a9840 con 0x55f7f4f22e40 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38836/0 (socket says 172.21.15.26:38836) 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 -- 172.21.15.26:0/641165972 learned_addr learned my addr 172.21.15.26:0/641165972 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd919640 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22e40 0x55f7f4f29d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 -- 172.21.15.26:0/641165972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22e40 msgr2=0x55f7f4f29d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22e40 0x55f7f4f29d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 -- 172.21.15.26:0/641165972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 msgr2=0x55f7f4f31020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f31020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 -- 172.21.15.26:0/641165972 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f4f22c70 con 0x55f7f4f22520 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0cd118640 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8022860 tx=0x7fc0b8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243a49a3cc889c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.227+0000 7fc0b37fe640 1 -- 172.21.15.26:0/641165972 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8005b90 con 0x55f7f4f22520 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0b37fe640 1 -- 172.21.15.26:0/641165972 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0b8005850 con 0x55f7f4f22520 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0b37fe640 1 -- 172.21.15.26:0/641165972 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8022440 con 0x55f7f4f22520 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- 172.21.15.26:0/641165972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 msgr2=0x55f7f4f3d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8022860 tx=0x7fc0b8004610 comp rx=0 tx=0).stop 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- 172.21.15.26:0/641165972 shutdown_connections 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f31020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22e40 0x55f7f4f29d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 --2- 172.21.15.26:0/641165972 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22520 0x55f7f4f3d6e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- 172.21.15.26:0/641165972 >> 172.21.15.26:0/641165972 conn(0x55f7f4f1eb30 msgr2=0x55f7f4f29670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- 172.21.15.26:0/641165972 shutdown_connections 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- 172.21.15.26:0/641165972 wait complete. 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.228+0000 7fc0d2974740 1 -- start start 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 Processor -- start 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f39de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81a9e90 con 0x55f7f4f22e40 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81ab300 con 0x55f7f4f31c10 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81aa560 con 0x55f7f4f22520 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0d2974740 10 client.0 _ll_register_callbacks cb 0x7fc0b8012920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38842/0 (socket says 172.21.15.26:38842) 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 -- 172.21.15.26:0/2882655370 learned_addr learned my addr 172.21.15.26:0/2882655370 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd118640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd118640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f39de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 -- 172.21.15.26:0/2882655370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 msgr2=0x55f7f4f39de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f39de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.229+0000 7fc0cd919640 1 -- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 msgr2=0x55f7f4f2b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0cd919640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0cd919640 1 -- 172.21.15.26:0/2882655370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f4f2bb80 con 0x55f7f4f22e40 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0cd118640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f39de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0cd919640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4032cd0 tx=0x7fc0c40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae07f35edfd8b62a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4054060 con 0x55f7f4f22e40 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0b8005850 con 0x55f7f4f22e40 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc0c4031430 con 0x55f7f4f22e40 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc0c403f040 con 0x55f7f4f22e40 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0d2974740 10 client.16761 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f7f4fa8030 con 0x55f7f4f22e40 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0b8005850 con 0x55f7f4f22e40 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7fc0c4004990 con 0x55f7f4f22e40 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0d2974740 10 client.16761 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.230+0000 7fc0d2974740 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0d2974740 20 client.16761 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4fa8370 0x55f7f4faa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0d2974740 4 client.16761 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7f4e8f200 con 0x55f7f4fa8370 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0cd118640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4fa8370 0x55f7f4faa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0b17fa640 10 client.16761 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.231+0000 7fc0cd118640 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4fa8370 0x55f7f4faa760 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceda25e66c774ccc server_cookie=a5ea74efef9f54c1 in_seq=0 out_seq=0 2024-04-23T18:26:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.244+0000 7fc0b17fa640 1 -- 172.21.15.26:0/2882655370 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7f4e8f200 con 0x55f7f4fa8370 2024-04-23T18:26:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0b17fa640 10 client.16761 handle_command_reply: tid=0 2024-04-23T18:26:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 client.16761 shutdown 2024-04-23T18:26:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 msgr2=0x55f7f4f2b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4032cd0 tx=0x7fc0c40058b0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 shutdown_connections 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55f7f4fa8370 0x55f7f4faa760 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f4f31c10 0x55f7f4f2b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55f7f4f22e40 0x55f7f4f2b050 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0b17fa640 0 client.16761 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 --2- 172.21.15.26:0/2882655370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f7f4f22520 0x55f7f4f39de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 >> 172.21.15.26:0/2882655370 conn(0x55f7f4f1eb30 msgr2=0x55f7f4f24070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 shutdown_connections 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 1 -- 172.21.15.26:0/2882655370 wait complete. 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.245+0000 7fc0d2974740 20 client.16761 trim_cache size 0 max 16384 2024-04-23T18:26:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:26:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:26:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:26:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:26:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 msgr2=0x7fc0b40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 0x7fc0b40a6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0099230 tx=0x7fc0c0087040 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 msgr2=0x7fc0c819cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000e960 tx=0x7fc0c000ee20 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 shutdown_connections 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fc0b40a3e40 0x7fc0b40a6300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0c8101b40 0x7fc0c8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fc0c8105e20 0x7fc0c819cf90 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 --2- 172.21.15.26:0/3035766369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0c81011a0 0x7fc0c81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 >> 172.21.15.26:0/3035766369 conn(0x7fc0c8076c50 msgr2=0x7fc0c806aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 shutdown_connections 2024-04-23T18:26:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:21.246+0000 7fc0cfb7d640 1 -- 172.21.15.26:0/3035766369 wait complete. 2024-04-23T18:26:21.242 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:20 smithi131 ceph-mon[33041]: pgmap v285: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-04-23T18:26:21.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.55 MB/sec execute 376 sec latency 19659.549 ms 2024-04-23T18:26:22.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.53 MB/sec execute 377 sec latency 20659.582 ms 2024-04-23T18:26:23.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:22 smithi110 ceph-mon[32585]: pgmap v286: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-04-23T18:26:23.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:22 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:23.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:22 smithi026 ceph-mon[29364]: pgmap v286: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-04-23T18:26:23.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:22 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:22 smithi131 ceph-mon[33041]: pgmap v286: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-04-23T18:26:23.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:22 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:23.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.52 MB/sec execute 378 sec latency 21659.612 ms 2024-04-23T18:26:24.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:23 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:24.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:23 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:24.191 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:23 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:26:24.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:23 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:24.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:23 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:24.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:23 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:26:24.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:23 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:24.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:23 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:24.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:23 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:26:24.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.50 MB/sec execute 379 sec latency 22659.662 ms 2024-04-23T18:26:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:25 smithi110 ceph-mon[32585]: pgmap v287: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:26:25.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:25 smithi026 ceph-mon[29364]: pgmap v287: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:26:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:25 smithi131 ceph-mon[33041]: pgmap v287: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:26:25.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540333 5.49 MB/sec execute 380 sec latency 23659.724 ms 2024-04-23T18:26:25.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 -- 172.21.15.26:0/1633188250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24106710 msgr2=0x7f3b24106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 --2- 172.21.15.26:0/1633188250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24106710 0x7f3b24106af0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009900 tx=0x7f3b1402a480 comp rx=0 tx=0).stop 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 -- 172.21.15.26:0/1633188250 shutdown_connections 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 --2- 172.21.15.26:0/1633188250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 0x7f3b24100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 --2- 172.21.15.26:0/1633188250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24101260 0x7f3b2410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 --2- 172.21.15.26:0/1633188250 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24106710 0x7f3b24106af0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 -- 172.21.15.26:0/1633188250 >> 172.21.15.26:0/1633188250 conn(0x7f3b24076130 msgr2=0x7f3b24076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 -- 172.21.15.26:0/1633188250 shutdown_connections 2024-04-23T18:26:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.817+0000 7f3b2912d640 1 -- 172.21.15.26:0/1633188250 wait complete. 2024-04-23T18:26:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.818+0000 7f3b2912d640 1 Processor -- start 2024-04-23T18:26:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.818+0000 7f3b2912d640 1 -- start start 2024-04-23T18:26:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 0x7f3b24198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 0x7f3b2419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2410ffa0 con 0x7f3b24101260 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2410fe20 con 0x7f3b241008a0 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24110120 con 0x7f3b24106710 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38850/0 (socket says 172.21.15.26:38850) 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 -- 172.21.15.26:0/719797503 learned_addr learned my addr 172.21.15.26:0/719797503 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2359e640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 0x7f3b2419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b22d9d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 0x7f3b24198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 -- 172.21.15.26:0/719797503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 msgr2=0x7f3b2419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 0x7f3b2419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 -- 172.21.15.26:0/719797503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 msgr2=0x7f3b24198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 0x7f3b24198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 -- 172.21.15.26:0/719797503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2419d650 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2359e640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 0x7f3b2419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2259c640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800ddc0 tx=0x7f3b1800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1801b070 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1800be40 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.819+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2419d940 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.820+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18016a80 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.820+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b241a51d0 con 0x7f3b24101260 2024-04-23T18:26:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.821+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b240691c0 con 0x7f3b24101260 2024-04-23T18:26:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.822+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b1801e070 con 0x7f3b24101260 2024-04-23T18:26:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.823+0000 7f3b0bfff640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 0x7f3b0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.823+0000 7f3b22d9d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 0x7f3b0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.823+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3b180d0510 con 0x7f3b24101260 2024-04-23T18:26:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.823+0000 7f3b22d9d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 0x7f3b0c0a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3b14031440 tx=0x7f3b140095f0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:25.828+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b1809ccc0 con 0x7f3b24101260 2024-04-23T18:26:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.057+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2402d5c0 con 0x7f3b24101260 2024-04-23T18:26:26.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.058+0000 7f3b0bfff640 1 -- 172.21.15.26:0/719797503 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11886 (secure 0 0 0) 0x7f3b1809c5c0 con 0x7f3b24101260 2024-04-23T18:26:26.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:26:26.051 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:26:09.422259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:26:26.051 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 45 2024-04-23T18:26:26.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 msgr2=0x7f3b0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 0x7f3b0c0a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3b14031440 tx=0x7f3b140095f0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 msgr2=0x7f3b24198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800ddc0 tx=0x7f3b1800a620 comp rx=0 tx=0).stop 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 shutdown_connections 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3b0c0a3d20 0x7f3b0c0a61e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b24106710 0x7f3b2419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3b24101260 0x7f3b24198ba0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 --2- 172.21.15.26:0/719797503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b241008a0 0x7f3b24198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 >> 172.21.15.26:0/719797503 conn(0x7f3b24076130 msgr2=0x7f3b241049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 shutdown_connections 2024-04-23T18:26:26.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.060+0000 7f3b2912d640 1 -- 172.21.15.26:0/719797503 wait complete. 2024-04-23T18:26:26.065 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 73106a2a --release --await' 2024-04-23T18:26:26.065 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 73106a2a --release --await 2024-04-23T18:26:26.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 -- 172.21.15.26:0/3331902114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 msgr2=0x7f3a2c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 --2- 172.21.15.26:0/3331902114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c101470 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009900 tx=0x7f3a1c02a410 comp rx=0 tx=0).stop 2024-04-23T18:26:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 -- 172.21.15.26:0/3331902114 shutdown_connections 2024-04-23T18:26:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 --2- 172.21.15.26:0/3331902114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c101a40 0x7f3a2c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 --2- 172.21.15.26:0/3331902114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c106160 0x7f3a2c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 --2- 172.21.15.26:0/3331902114 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c101470 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 -- 172.21.15.26:0/3331902114 >> 172.21.15.26:0/3331902114 conn(0x7f3a2c077f50 msgr2=0x7f3a2c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 -- 172.21.15.26:0/3331902114 shutdown_connections 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 -- 172.21.15.26:0/3331902114 wait complete. 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.187+0000 7f3a30cae640 1 Processor -- start 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 -- start start 2024-04-23T18:26:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 0x7f3a2c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c19cfd0 0x7f3a2c19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c112210 con 0x7f3a2c101090 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c112090 con 0x7f3a2c106160 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a30cae640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c112390 con 0x7f3a2c19cfd0 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38874/0 (socket says 172.21.15.26:38874) 2024-04-23T18:26:26.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 -- 172.21.15.26:0/1798840535 learned_addr learned my addr 172.21.15.26:0/1798840535 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c19cfd0 0x7f3a2c19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 0x7f3a2c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 -- 172.21.15.26:0/1798840535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c19cfd0 msgr2=0x7f3a2c19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c19cfd0 0x7f3a2c19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 -- 172.21.15.26:0/1798840535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 msgr2=0x7f3a2c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 0x7f3a2c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 -- 172.21.15.26:0/1798840535 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c19dac0 con 0x7f3a2c101090 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 0x7f3a2c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.188+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c005ab0 tx=0x7f3a1c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.189+0000 7f3a1b7fe640 1 -- 172.21.15.26:0/1798840535 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c03f070 con 0x7f3a2c101090 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.189+0000 7f3a1b7fe640 1 -- 172.21.15.26:0/1798840535 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a1c031d60 con 0x7f3a2c101090 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.189+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c19dd50 con 0x7f3a2c101090 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.189+0000 7f3a1b7fe640 1 -- 172.21.15.26:0/1798840535 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c035030 con 0x7f3a2c101090 2024-04-23T18:26:26.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.189+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c1a5510 con 0x7f3a2c101090 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.191+0000 7f3a1b7fe640 1 -- 172.21.15.26:0/1798840535 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3a1c042070 con 0x7f3a2c101090 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.191+0000 7f3a339c8740 1 Processor -- start 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.191+0000 7f3a339c8740 1 -- start start 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a1b7fe640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b562870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b4c76d0 con 0x55bc0b56d480 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b4641a0 con 0x55bc0b562490 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a339c8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b4c3690 con 0x55bc0b562db0 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a1b7fe640 1 -- 172.21.15.26:0/1798840535 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a1c0f11e0 con 0x7f3a2c101090 2024-04-23T18:26:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38888/0 (socket says 172.21.15.26:38888) 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3276824024 learned_addr learned my addr 172.21.15.26:0/3276824024 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b562870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3276824024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 msgr2=0x55bc0b56cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3276824024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 msgr2=0x55bc0b562870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b562870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3276824024 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0b46e5b0 con 0x55bc0b56d480 2024-04-23T18:26:26.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 0x7f3a140a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a240077f0 tx=0x7f3a24007e40 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.192+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3a24011c40 tx=0x7f3a2401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c9d20f5cd70d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a18ff9640 1 -- 172.21.15.26:0/3276824024 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24022070 con 0x55bc0b56d480 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a18ff9640 1 -- 172.21.15.26:0/3276824024 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a2400fe70 con 0x55bc0b56d480 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a18ff9640 1 -- 172.21.15.26:0/3276824024 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240054b0 con 0x55bc0b56d480 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 -- 172.21.15.26:0/3276824024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 msgr2=0x55bc0b56f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3a24011c40 tx=0x7f3a2401b000 comp rx=0 tx=0).stop 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 -- 172.21.15.26:0/3276824024 shutdown_connections 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56f870 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3276824024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b562870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 -- 172.21.15.26:0/3276824024 >> 172.21.15.26:0/3276824024 conn(0x55bc0b55e7b0 msgr2=0x55bc0b55eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 -- 172.21.15.26:0/3276824024 shutdown_connections 2024-04-23T18:26:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.193+0000 7f3a339c8740 1 -- 172.21.15.26:0/3276824024 wait complete. 2024-04-23T18:26:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 -- start start 2024-04-23T18:26:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 Processor -- start 2024-04-23T18:26:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562490 0x55bc0b56ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 0x55bc0b56a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b5702b0 con 0x55bc0b562db0 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b461b20 con 0x55bc0b56d480 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55bc0b570430 con 0x55bc0b562490 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a339c8740 10 client.0 _ll_register_callbacks cb 0x55bc0b461530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38900/0 (socket says 172.21.15.26:38900) 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 -- 172.21.15.26:0/630428977 learned_addr learned my addr 172.21.15.26:0/630428977 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562490 0x55bc0b56ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 0x55bc0b56a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 -- 172.21.15.26:0/630428977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562490 msgr2=0x55bc0b56ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562490 0x55bc0b56ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 -- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 msgr2=0x55bc0b56a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 0x55bc0b56a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.194+0000 7f3a29d9b640 1 -- 172.21.15.26:0/630428977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0b1d9920 con 0x55bc0b562db0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 0x55bc0b56a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3a20010690 tx=0x7f3a20010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed895c8ff44fea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2001d060 con 0x55bc0b562db0 2024-04-23T18:26:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a20018430 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a2400fe70 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3a2000ee60 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 10 client.16779 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55bc0b2a5d30 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a2400fe70 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f3a2000d450 con 0x55bc0b562db0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 10 client.16779 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 10 client.16779 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 20 client.16779 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b57d7b0 0x55bc0b57fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.195+0000 7f3a339c8740 4 client.16779 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.196+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc0b485500 con 0x55bc0b57d7b0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.196+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b57d7b0 0x55bc0b57fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.196+0000 7f3a11ffb640 10 client.16779 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.196+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b57d7b0 0x55bc0b57fbd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0c8865e56dfaaf server_cookie=c21affc56cb31589 in_seq=0 out_seq=0 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a11ffb640 1 -- 172.21.15.26:0/630428977 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc0b485500 con 0x55bc0b57d7b0 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a11ffb640 10 client.16779 handle_command_reply: tid=0 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 client.16779 shutdown 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 msgr2=0x55bc0b56a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3a20010690 tx=0x7f3a20010b50 comp rx=0 tx=0).stop 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 shutdown_connections 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b57d7b0 0x55bc0b57fbd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b56d480 0x55bc0b56a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a11ffb640 0 client.16779 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b562db0 0x55bc0b56a210 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 --2- 172.21.15.26:0/630428977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562490 0x55bc0b56ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 >> 172.21.15.26:0/630428977 conn(0x55bc0b55e7b0 msgr2=0x55bc0b5645b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 shutdown_connections 2024-04-23T18:26:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.197+0000 7f3a339c8740 1 -- 172.21.15.26:0/630428977 wait complete. 2024-04-23T18:26:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.198+0000 7f3a339c8740 20 client.16779 trim_cache size 0 max 16384 2024-04-23T18:26:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 Processor -- start 2024-04-23T18:26:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 -- start start 2024-04-23T18:26:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b585480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1b03d0 con 0x55bc0b56d480 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1adc70 con 0x55bc0b562490 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a339c8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1abae0 con 0x55bc0b562db0 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38910/0 (socket says 172.21.15.26:38910) 2024-04-23T18:26:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3813844101 learned_addr learned my addr 172.21.15.26:0/3813844101 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3813844101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 msgr2=0x55bc0b585480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b585480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3813844101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 msgr2=0x55bc0b5791f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5791f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3813844101 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0b55e5e0 con 0x55bc0b56d480 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.215+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5791f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3a2400fc30 tx=0x7f3a2400fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0685f6d073802b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a13fff640 1 -- 172.21.15.26:0/3813844101 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24022040 con 0x55bc0b56d480 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a13fff640 1 -- 172.21.15.26:0/3813844101 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a240140c0 con 0x55bc0b56d480 2024-04-23T18:26:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a13fff640 1 -- 172.21.15.26:0/3813844101 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24012c30 con 0x55bc0b56d480 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 -- 172.21.15.26:0/3813844101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 msgr2=0x55bc0b5859c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3a2400fc30 tx=0x7f3a2400fe70 comp rx=0 tx=0).stop 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 -- 172.21.15.26:0/3813844101 shutdown_connections 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b5859c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b585480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3813844101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5791f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 -- 172.21.15.26:0/3813844101 >> 172.21.15.26:0/3813844101 conn(0x55bc0b55e7b0 msgr2=0x55bc0b56c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 -- 172.21.15.26:0/3813844101 shutdown_connections 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.216+0000 7f3a339c8740 1 -- 172.21.15.26:0/3813844101 wait complete. 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 -- start start 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 Processor -- start 2024-04-23T18:26:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1ac130 con 0x55bc0b56d480 2024-04-23T18:26:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1ad5a0 con 0x55bc0b562490 2024-04-23T18:26:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1ac800 con 0x55bc0b562db0 2024-04-23T18:26:26.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a339c8740 10 client.0 _ll_register_callbacks cb 0x7f3a2000be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:26.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38926/0 (socket says 172.21.15.26:38926) 2024-04-23T18:26:26.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3014319748 learned_addr learned my addr 172.21.15.26:0/3014319748 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a29d9b640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3014319748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 msgr2=0x55bc0b56dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 msgr2=0x55bc0b5759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 -- 172.21.15.26:0/3014319748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0b56eb90 con 0x55bc0b56d480 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a2ad9d640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3a24014d50 tx=0x7f3a24014d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c59f36d79980d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.217+0000 7f3a11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24022070 con 0x55bc0b56d480 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a240140c0 con 0x55bc0b56d480 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a24004df0 con 0x55bc0b56d480 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3a24012870 con 0x55bc0b56d480 2024-04-23T18:26:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 10 client.16785 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55bc0b5e3cf0 con 0x55bc0b56d480 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a240140c0 con 0x55bc0b56d480 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f3a2402d020 con 0x55bc0b56d480 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 10 client.16785 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 10 client.16785 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 20 client.16785 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b5e4220 0x55bc0b5e6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 4 client.16785 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "73106a2a", "release": true, "await": true}] 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "73106a2a", "release": true, "await": true}) -- 0x55bc0b4d50f0 con 0x55bc0b5e4220 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.218+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b5e4220 0x55bc0b5e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.219+0000 7f3a2a59c640 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b5e4220 0x55bc0b5e6610 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f8360c1d67fad3e server_cookie=98cf52c4c34b243c in_seq=0 out_seq=0 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.219+0000 7f3a11ffb640 10 client.16785 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a11ffb640 1 -- 172.21.15.26:0/3014319748 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55bc0b4d50f0 con 0x55bc0b5e4220 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a11ffb640 10 client.16785 handle_command_reply: tid=0 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 client.16785 shutdown 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 msgr2=0x55bc0b56e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3a24014d50 tx=0x7f3a24014d80 comp rx=0 tx=0).stop 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 shutdown_connections 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55bc0b5e4220 0x55bc0b5e6610 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55bc0b56d480 0x55bc0b56e420 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bc0b562db0 0x55bc0b56dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a11ffb640 0 client.16785 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 --2- 172.21.15.26:0/3014319748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc0b562490 0x55bc0b5759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 >> 172.21.15.26:0/3014319748 conn(0x55bc0b55e7b0 msgr2=0x55bc0b563c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 shutdown_connections 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.223+0000 7f3a339c8740 1 -- 172.21.15.26:0/3014319748 wait complete. 2024-04-23T18:26:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a339c8740 20 client.16785 trim_cache size 0 max 16384 2024-04-23T18:26:26.234 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:26:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 45, 2024-04-23T18:26:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:26:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 40, 2024-04-23T18:26:26.234 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "73106a2a": { 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 40, 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:26:26.235 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 83.7, 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:26:26.236 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.238 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 msgr2=0x7f3a140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 0x7f3a140a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a240077f0 tx=0x7f3a24007e40 comp rx=0 tx=0).stop 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 msgr2=0x7f3a2c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c005ab0 tx=0x7f3a1c003d50 comp rx=0 tx=0).stop 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 shutdown_connections 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a2c106160 0x7f3a2c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a2c19cfd0 0x7f3a2c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 --2- 172.21.15.26:0/1798840535 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3a2c101090 0x7f3a2c198710 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 >> 172.21.15.26:0/1798840535 conn(0x7f3a2c077f50 msgr2=0x7f3a2c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 shutdown_connections 2024-04-23T18:26:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.224+0000 7f3a30cae640 1 -- 172.21.15.26:0/1798840535 wait complete. 2024-04-23T18:26:26.242 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '73106a2a', seconds elapsed: 24.8 2024-04-23T18:26:26.242 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.65 2024-04-23T18:26:26.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:26.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 -- 172.21.15.26:0/595817630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68106160 msgr2=0x7efd6810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/595817630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68106160 0x7efd6810dd30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efd60009ca0 tx=0x7efd6002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 -- 172.21.15.26:0/595817630 shutdown_connections 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/595817630 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68106160 0x7efd6810dd30 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/595817630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd68101a40 0x7efd68105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/595817630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68101090 0x7efd68101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 -- 172.21.15.26:0/595817630 >> 172.21.15.26:0/595817630 conn(0x7efd680771d0 msgr2=0x7efd680775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 -- 172.21.15.26:0/595817630 shutdown_connections 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.358+0000 7efd6e6cb640 1 -- 172.21.15.26:0/595817630 wait complete. 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 Processor -- start 2024-04-23T18:26:26.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 -- start start 2024-04-23T18:26:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 0x7efd68103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd681035a0 0x7efd681a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7efd681110d0 con 0x7efd68101090 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efd68110f50 con 0x7efd68106160 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6e6cb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7efd68111250 con 0x7efd681035a0 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd67fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd67fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38938/0 (socket says 172.21.15.26:38938) 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd67fff640 1 -- 172.21.15.26:0/3189406635 learned_addr learned my addr 172.21.15.26:0/3189406635 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd677fe640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 0x7efd68103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.359+0000 7efd6cc68640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd681035a0 0x7efd681a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 -- 172.21.15.26:0/3189406635 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd681035a0 msgr2=0x7efd681a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd681035a0 0x7efd681a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 -- 172.21.15.26:0/3189406635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 msgr2=0x7efd68103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 0x7efd68103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 -- 172.21.15.26:0/3189406635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd681a6fe0 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd677fe640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 0x7efd68103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd67fff640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efd5800ea60 tx=0x7efd5800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5800cde0 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd580035f0 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd681a72d0 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58010690 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.360+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd681a7820 con 0x7efd68101090 2024-04-23T18:26:26.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.361+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd68102560 con 0x7efd68101090 2024-04-23T18:26:26.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.363+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efd580037a0 con 0x7efd68101090 2024-04-23T18:26:26.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.364+0000 7efd657fa640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 0x7efd500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.364+0000 7efd677fe640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 0x7efd500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.364+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7efd580d0cc0 con 0x7efd68101090 2024-04-23T18:26:26.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.364+0000 7efd677fe640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 0x7efd500a6210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd681045b0 tx=0x7efd5c005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.368+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd58053ff0 con 0x7efd68101090 2024-04-23T18:26:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.604+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd680691c0 con 0x7efd68101090 2024-04-23T18:26:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.605+0000 7efd657fa640 1 -- 172.21.15.26:0/3189406635 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11886 (secure 0 0 0) 0x7efd680691c0 con 0x7efd68101090 2024-04-23T18:26:26.598 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:26:26.598 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:26:09.422259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24464,24503,24460]},"id":1}]} 2024-04-23T18:26:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 45 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 msgr2=0x7efd500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 0x7efd500a6210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd681045b0 tx=0x7efd5c005f90 comp rx=0 tx=0).stop 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 msgr2=0x7efd68102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efd5800ea60 tx=0x7efd5800ef20 comp rx=0 tx=0).stop 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 shutdown_connections 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7efd500a3d50 0x7efd500a6210 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd68106160 0x7efd68103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efd681035a0 0x7efd681a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 --2- 172.21.15.26:0/3189406635 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7efd68101090 0x7efd68102b20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 >> 172.21.15.26:0/3189406635 conn(0x7efd680771d0 msgr2=0x7efd680ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 shutdown_connections 2024-04-23T18:26:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.607+0000 7efd6e6cb640 1 -- 172.21.15.26:0/3189406635 wait complete. 2024-04-23T18:26:26.611 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 60 --expiration 83.65 --await-for 42.0' 2024-04-23T18:26:26.611 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 60 --expiration 83.65 --await-for 42.0 2024-04-23T18:26:26.689 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540712 5.48 MB/sec execute 381 sec latency 24195.043 ms 2024-04-23T18:26:26.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 -- 172.21.15.26:0/3651591977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406b290 msgr2=0x7f5324106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 --2- 172.21.15.26:0/3651591977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406b290 0x7f5324106b90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5320009900 tx=0x7f532002a410 comp rx=0 tx=0).stop 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 -- 172.21.15.26:0/3651591977 shutdown_connections 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 --2- 172.21.15.26:0/3651591977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 0x7f532410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 --2- 172.21.15.26:0/3651591977 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406b290 0x7f5324106b90 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 --2- 172.21.15.26:0/3651591977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532406a8e0 0x7f532406acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 -- 172.21.15.26:0/3651591977 >> 172.21.15.26:0/3651591977 conn(0x7f5324077f20 msgr2=0x7f5324078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 -- 172.21.15.26:0/3651591977 shutdown_connections 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.733+0000 7f532c144640 1 -- 172.21.15.26:0/3651591977 wait complete. 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 Processor -- start 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 -- start start 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 0x7f5324198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532419cfb0 0x7f532419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f53241121b0 con 0x7f532406a8e0 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5324112030 con 0x7f532419cfb0 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532c144640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5324112330 con 0x7f5324107270 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38954/0 (socket says 172.21.15.26:38954) 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f5329ee0640 1 -- 172.21.15.26:0/35747242 learned_addr learned my addr 172.21.15.26:0/35747242 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f53296df640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 0x7f5324198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.734+0000 7f532a6e1640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532419cfb0 0x7f532419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 -- 172.21.15.26:0/35747242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 msgr2=0x7f5324198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 0x7f5324198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 -- 172.21.15.26:0/35747242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532419cfb0 msgr2=0x7f532419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532419cfb0 0x7f532419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 -- 172.21.15.26:0/35747242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532419daa0 con 0x7f532406a8e0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f5329ee0640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f531400ea60 tx=0x7f531400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f531affd640 1 -- 172.21.15.26:0/35747242 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400cde0 con 0x7f532406a8e0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f531affd640 1 -- 172.21.15.26:0/35747242 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53140035f0 con 0x7f532406a8e0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f532419dd90 con 0x7f532406a8e0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f531affd640 1 -- 172.21.15.26:0/35747242 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314010690 con 0x7f532406a8e0 2024-04-23T18:26:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.735+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241a55a0 con 0x7f532406a8e0 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.738+0000 7f531affd640 1 -- 172.21.15.26:0/35747242 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f53140037a0 con 0x7f532406a8e0 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.738+0000 7f532ee69740 1 Processor -- start 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.738+0000 7f532ee69740 1 -- start start 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56469fafe370 con 0x56469fb990a0 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56469fa9b2e0 con 0x56469fba4090 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532ee69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56469fafa330 con 0x56469fb999c0 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38956/0 (socket says 172.21.15.26:38956) 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f53296df640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 -- 172.21.15.26:0/534142527 learned_addr learned my addr 172.21.15.26:0/534142527 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:53836/0 (socket says 172.21.15.26:53836) 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f531affd640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 0x7f53100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 -- 172.21.15.26:0/534142527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 msgr2=0x56469fba3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 -- 172.21.15.26:0/534142527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 msgr2=0x56469fba6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 -- 172.21.15.26:0/534142527 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56469f8dcd80 con 0x56469fb990a0 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f53296df640 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f531affd640 1 -- 172.21.15.26:0/35747242 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f53140d0a20 con 0x7f532406a8e0 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.739+0000 7f532a6e1640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 0x7f53100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.740+0000 7f532a6e1640 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f531c009f50 tx=0x7f531c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c35cbbb4ece45b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.740+0000 7f530ffff640 1 -- 172.21.15.26:0/534142527 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c004250 con 0x56469fb990a0 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.740+0000 7f530ffff640 1 -- 172.21.15.26:0/534142527 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f531c00fe00 con 0x56469fb990a0 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.740+0000 7f530ffff640 1 -- 172.21.15.26:0/534142527 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c01b9d0 con 0x56469fb990a0 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- 172.21.15.26:0/534142527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 msgr2=0x56469fb99480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f531c009f50 tx=0x7f531c0118d0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532a6e1640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 0x7f53100a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f531c012fd0 tx=0x7f531c021000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- 172.21.15.26:0/534142527 shutdown_connections 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- 172.21.15.26:0/534142527 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb990a0 0x56469fb99480 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- 172.21.15.26:0/534142527 >> 172.21.15.26:0/534142527 conn(0x56469fb953c0 msgr2=0x56469fb957a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- 172.21.15.26:0/534142527 shutdown_connections 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- 172.21.15.26:0/534142527 wait complete. 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- start start 2024-04-23T18:26:26.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 Processor -- start 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 0x56469fba1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.741+0000 7f532ee69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x56469fba6e80 con 0x56469fb999c0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f532ee69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56469fa98fb0 con 0x56469fba4090 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f532ee69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x56469fba7000 con 0x56469fb990a0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f532ee69740 10 client.0 _ll_register_callbacks cb 0x56469fa989a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38970/0 (socket says 172.21.15.26:38970) 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 -- 172.21.15.26:0/4128895912 learned_addr learned my addr 172.21.15.26:0/4128895912 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f532a6e1640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f5329ee0640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 0x56469fba1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 -- 172.21.15.26:0/4128895912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 msgr2=0x56469fba1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 0x56469fba1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 -- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 msgr2=0x56469fba1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 -- 172.21.15.26:0/4128895912 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56469f93d510 con 0x56469fb999c0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f5329ee0640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 0x56469fba1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f53296df640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5320033000 tx=0x7f5320032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1770261fa9edbef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f530d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320035040 con 0x56469fb999c0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f5320004050 con 0x56469fb999c0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.742+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f531c00fe00 con 0x56469fb999c0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5320004250 con 0x56469fb999c0 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 10 client.16803 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56469f7d0670 con 0x56469fb999c0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f531c00fe00 con 0x56469fb999c0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f5320004a50 con 0x56469fb999c0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 10 client.16803 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 10 client.16803 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 20 client.16803 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fbb4360 0x56469fbb6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 4 client.16803 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56469fb0bd90 con 0x56469fbb4360 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.743+0000 7f5329ee0640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fbb4360 0x56469fbb6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.744+0000 7f5329ee0640 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fbb4360 0x56469fbb6790 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3326db92ab98279d server_cookie=a42626c878e28506 in_seq=0 out_seq=0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.744+0000 7f530d7fa640 10 client.16803 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.744+0000 7f530d7fa640 1 -- 172.21.15.26:0/4128895912 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56469fb0bd90 con 0x56469fbb4360 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.744+0000 7f530d7fa640 10 client.16803 handle_command_reply: tid=0 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 client.16803 shutdown 2024-04-23T18:26:26.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 msgr2=0x56469fba1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5320033000 tx=0x7f5320032f20 comp rx=0 tx=0).stop 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 shutdown_connections 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fbb4360 0x56469fbb6790 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fba4090 0x56469fba1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fb999c0 0x56469fba1560 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 --2- 172.21.15.26:0/4128895912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb990a0 0x56469fba1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f530d7fa640 0 client.16803 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 >> 172.21.15.26:0/4128895912 conn(0x56469fb953c0 msgr2=0x56469fb9b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 shutdown_connections 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 1 -- 172.21.15.26:0/4128895912 wait complete. 2024-04-23T18:26:26.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.745+0000 7f532ee69740 20 client.16803 trim_cache size 0 max 16384 2024-04-23T18:26:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 Processor -- start 2024-04-23T18:26:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 -- start start 2024-04-23T18:26:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fb943e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fbafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fbbc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b0470 con 0x56469fba4090 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53241add10 con 0x56469fb990a0 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532ee69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f53241abb80 con 0x56469fb999c0 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532a6e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fbbc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532a6e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fbbc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38980/0 (socket says 172.21.15.26:38980) 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f532a6e1640 1 -- 172.21.15.26:0/3355305177 learned_addr learned my addr 172.21.15.26:0/3355305177 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fb943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f53296df640 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fbafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 -- 172.21.15.26:0/3355305177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 msgr2=0x56469fbafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fbafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 -- 172.21.15.26:0/3355305177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 msgr2=0x56469fbbc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fbbc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.763+0000 7f5329ee0640 1 -- 172.21.15.26:0/3355305177 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56469fb90ca0 con 0x56469fb990a0 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f5329ee0640 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fb943e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f53140ce170 tx=0x7f53140bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fff6381efbfb2a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f530f7fe640 1 -- 172.21.15.26:0/3355305177 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531409c500 con 0x56469fb990a0 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f530f7fe640 1 -- 172.21.15.26:0/3355305177 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53140d9040 con 0x56469fb990a0 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f530f7fe640 1 -- 172.21.15.26:0/3355305177 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531409c320 con 0x56469fb990a0 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 -- 172.21.15.26:0/3355305177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 msgr2=0x56469fb943e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fb943e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f53140ce170 tx=0x7f53140bd040 comp rx=0 tx=0).stop 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 -- 172.21.15.26:0/3355305177 shutdown_connections 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fbbc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fbafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 --2- 172.21.15.26:0/3355305177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fb943e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.764+0000 7f532ee69740 1 -- 172.21.15.26:0/3355305177 >> 172.21.15.26:0/3355305177 conn(0x56469fb953c0 msgr2=0x56469fba1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- 172.21.15.26:0/3355305177 shutdown_connections 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- 172.21.15.26:0/3355305177 wait complete. 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- start start 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 Processor -- start 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fba9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ac1d0 con 0x56469fba4090 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ad640 con 0x56469fb990a0 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ac8a0 con 0x56469fb999c0 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532ee69740 10 client.0 _ll_register_callbacks cb 0x7f5320032bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f532a6e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fba9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f5329ee0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:41556/0 (socket says 172.21.15.26:41556) 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f53296df640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.765+0000 7f5329ee0640 1 -- 172.21.15.26:0/2234479800 learned_addr learned my addr 172.21.15.26:0/2234479800 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 -- 172.21.15.26:0/2234479800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 msgr2=0x56469fba9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 -- 172.21.15.26:0/2234479800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 msgr2=0x56469fba9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fba9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 -- 172.21.15.26:0/2234479800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56469fc1a8e0 con 0x56469fb990a0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f532a6e1640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fba9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f5329ee0640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5314009920 tx=0x7f53140d4540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a96fa7e31e9afa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f530d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140b5070 con 0x56469fb990a0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53140d9040 con 0x56469fb990a0 2024-04-23T18:26:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f53140ce3f0 con 0x56469fb990a0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f531409c320 con 0x56469fb990a0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f532ee69740 10 client.26162 fetch_fsmap learned FSMap version 45 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.766+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56469fc1bb20 con 0x56469fb990a0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53140d9040 con 0x56469fb990a0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 45) ==== 8346+0+0 (secure 0 0 0) 0x7f53140bbb80 con 0x56469fb990a0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 10 client.26162 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 10 client.26162 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 20 client.26162 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fc1c1b0 0x56469fc1e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 4 client.26162 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.65, "await_for": 42.0}] 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.65, "await_for": 42.0}) -- 0x56469fb05c00 con 0x56469fc1c1b0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f53296df640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fc1c1b0 0x56469fc1e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f53296df640 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fc1c1b0 0x56469fc1e5d0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=251f532e429f3eac server_cookie=37b469e4ec5787a2 in_seq=0 out_seq=0 2024-04-23T18:26:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.767+0000 7f530d7fa640 10 client.26162 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f530d7fa640 1 -- 172.21.15.26:0/2234479800 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56469fb05c00 con 0x56469fc1c1b0 2024-04-23T18:26:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f530d7fa640 10 client.26162 handle_command_reply: tid=0 2024-04-23T18:26:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 client.26162 shutdown 2024-04-23T18:26:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 msgr2=0x56469fba9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5314009920 tx=0x7f53140d4540 comp rx=0 tx=0).stop 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 shutdown_connections 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x56469fc1c1b0 0x56469fc1e5d0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x56469fba4090 0x56469fba9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f530d7fa640 0 client.26162 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56469fb999c0 0x56469fba9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 --2- 172.21.15.26:0/2234479800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56469fb990a0 0x56469fba9220 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 >> 172.21.15.26:0/2234479800 conn(0x56469fb953c0 msgr2=0x56469fb9a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 shutdown_connections 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 1 -- 172.21.15.26:0/2234479800 wait complete. 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.821+0000 7f532ee69740 20 client.26162 trim_cache size 0 max 16384 2024-04-23T18:26:26.815 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 45, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 42, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "47cc512": { 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 42, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 83.7, 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:26:26.816 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "QUIESCED", 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:26.817 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 msgr2=0x7f53100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 0x7f53100a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f531c012fd0 tx=0x7f531c021000 comp rx=0 tx=0).stop 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 msgr2=0x7f53241986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f531400ea60 tx=0x7f531400ef20 comp rx=0 tx=0).stop 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 shutdown_connections 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f53100a3da0 0x7f53100a6260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532419cfb0 0x7f532419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5324107270 0x7f5324198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 --2- 172.21.15.26:0/35747242 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f532406a8e0 0x7f53241986f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 >> 172.21.15.26:0/35747242 conn(0x7f5324077f20 msgr2=0x7f53240ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 shutdown_connections 2024-04-23T18:26:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:26.822+0000 7f532c144640 1 -- 172.21.15.26:0/35747242 wait complete. 2024-04-23T18:26:26.825 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '47cc512', quiesce took 0.6 seconds. Will release after: 23.65 2024-04-23T18:26:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:27 smithi110 ceph-mon[32585]: pgmap v288: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-04-23T18:26:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/719797503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:27 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3189406635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:27 smithi026 ceph-mon[29364]: pgmap v288: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-04-23T18:26:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/719797503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:27 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3189406635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:27 smithi131 ceph-mon[33041]: pgmap v288: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 2.1 MiB/s wr, 90 op/s 2024-04-23T18:26:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/719797503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:27 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3189406635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:27.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.46 MB/sec execute 382 sec latency 895.098 ms 2024-04-23T18:26:28.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.45 MB/sec execute 383 sec latency 1895.208 ms 2024-04-23T18:26:29.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:29 smithi110 ceph-mon[32585]: pgmap v289: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:29.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:29 smithi026 ceph-mon[29364]: pgmap v289: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:29.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:29 smithi131 ceph-mon[33041]: pgmap v289: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:29.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.44 MB/sec execute 384 sec latency 2895.330 ms 2024-04-23T18:26:30.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.42 MB/sec execute 385 sec latency 3895.441 ms 2024-04-23T18:26:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:31 smithi110 ceph-mon[32585]: pgmap v290: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:31.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:31 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:31 smithi026 ceph-mon[29364]: pgmap v290: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:31.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:31 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:31 smithi131 ceph-mon[33041]: pgmap v290: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-04-23T18:26:31.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:31 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:31.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.41 MB/sec execute 386 sec latency 4895.554 ms 2024-04-23T18:26:32.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.39 MB/sec execute 387 sec latency 5895.698 ms 2024-04-23T18:26:33.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:33 smithi110 ceph-mon[32585]: pgmap v291: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-04-23T18:26:33.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:33 smithi026 ceph-mon[29364]: pgmap v291: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-04-23T18:26:33.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:33 smithi131 ceph-mon[33041]: pgmap v291: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-04-23T18:26:33.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.38 MB/sec execute 388 sec latency 6895.811 ms 2024-04-23T18:26:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 -- 172.21.15.26:0/3913036281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1010e790 msgr2=0x7f8d10110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3913036281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1010e790 0x7f8d10110be0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009ca0 tx=0x7f8d0002a950 comp rx=0 tx=0).stop 2024-04-23T18:26:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 -- 172.21.15.26:0/3913036281 shutdown_connections 2024-04-23T18:26:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3913036281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 0x7f8d1010e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3913036281 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d1010e790 0x7f8d10110be0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3913036281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d10074380 0x7f8d10074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 -- 172.21.15.26:0/3913036281 >> 172.21.15.26:0/3913036281 conn(0x7f8d100fda20 msgr2=0x7f8d100ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 -- 172.21.15.26:0/3913036281 shutdown_connections 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 -- 172.21.15.26:0/3913036281 wait complete. 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.345+0000 7f8d17e79640 1 Processor -- start 2024-04-23T18:26:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 -- start start 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 0x7f8d101a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1010e790 0x7f8d1019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101143c0 con 0x7f8d10074380 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10114240 con 0x7f8d10074ca0 2024-04-23T18:26:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d17e79640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10114540 con 0x7f8d1010e790 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38622/0 (socket says 172.21.15.26:38622) 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 -- 172.21.15.26:0/3814305531 learned_addr learned my addr 172.21.15.26:0/3814305531 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15414640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 0x7f8d101a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d16416640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1010e790 0x7f8d1019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 -- 172.21.15.26:0/3814305531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1010e790 msgr2=0x7f8d1019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1010e790 0x7f8d1019a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 -- 172.21.15.26:0/3814305531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 msgr2=0x7f8d101a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 0x7f8d101a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 -- 172.21.15.26:0/3814305531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1019af40 con 0x7f8d10074380 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.346+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800c970 tx=0x7f8d0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.347+0000 7f8d0effd640 1 -- 172.21.15.26:0/3814305531 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08007bb0 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.347+0000 7f8d0effd640 1 -- 172.21.15.26:0/3814305531 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d08007d50 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.347+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1019b230 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.347+0000 7f8d0effd640 1 -- 172.21.15.26:0/3814305531 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800f450 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.347+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d101a7540 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.349+0000 7f8d0effd640 1 -- 172.21.15.26:0/3814305531 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d08004f20 con 0x7f8d10074380 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.349+0000 7f8d1ab8f740 1 Processor -- start 2024-04-23T18:26:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 -- start start 2024-04-23T18:26:34.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976620 0x55a69e976a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.352 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e983bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a69e8db970 con 0x55a69e976f40 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a69e8784a0 con 0x55a69e981800 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a69e8d7930 con 0x55a69e976620 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d0effd640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 0x7f8cf80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38624/0 (socket says 172.21.15.26:38624) 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d15c15640 1 -- 172.21.15.26:0/3293498270 learned_addr learned my addr 172.21.15.26:0/3293498270 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d16416640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 0x7f8cf80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d16416640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976620 0x55a69e976a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d0effd640 1 -- 172.21.15.26:0/3814305531 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f8d080d0610 con 0x7f8d10074380 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.350+0000 7f8d15414640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e983bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 -- 172.21.15.26:0/3293498270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976620 msgr2=0x55a69e976a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976620 0x55a69e976a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 -- 172.21.15.26:0/3293498270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 msgr2=0x55a69e983bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e983bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 -- 172.21.15.26:0/3293498270 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69e5ee210 con 0x55a69e976f40 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15414640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e983bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d15c15640 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8d080cde10 tx=0x7f8d08098210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483f2f1b375fbf82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8d16416640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 0x7f8cf80a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d00005e30 tx=0x7f8d000098c0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8cf7fff640 1 -- 172.21.15.26:0/3293498270 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0801d070 con 0x55a69e976f40 2024-04-23T18:26:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8cf7fff640 1 -- 172.21.15.26:0/3293498270 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0809c5b0 con 0x55a69e976f40 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.351+0000 7f8cf7fff640 1 -- 172.21.15.26:0/3293498270 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08098c20 con 0x55a69e976f40 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/3293498270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 msgr2=0x55a69e9812c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8d080cde10 tx=0x7f8d08098210 comp rx=0 tx=0).stop 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/3293498270 shutdown_connections 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e983bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e9812c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/3293498270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976620 0x55a69e976a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/3293498270 >> 172.21.15.26:0/3293498270 conn(0x55a69e972b30 msgr2=0x55a69e972f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/3293498270 shutdown_connections 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/3293498270 wait complete. 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 -- start start 2024-04-23T18:26:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.352+0000 7f8d1ab8f740 1 Processor -- start 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976f40 0x55a69e97e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e97e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55a69e984430 con 0x55a69e976620 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a69e8760b0 con 0x55a69e981800 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55a69e9845b0 con 0x55a69e976f40 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d1ab8f740 10 client.0 _ll_register_callbacks cb 0x55a69e863d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38628/0 (socket says 172.21.15.26:38628) 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 -- 172.21.15.26:0/4071803624 learned_addr learned my addr 172.21.15.26:0/4071803624 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d16416640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e97e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15414640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976f40 0x55a69e97e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 -- 172.21.15.26:0/4071803624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976f40 msgr2=0x55a69e97e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976f40 0x55a69e97e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 -- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 msgr2=0x55a69e97e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e97e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 -- 172.21.15.26:0/4071803624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69e6ba620 con 0x55a69e976620 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8d15c15640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8d0809c540 tx=0x7f8d080870c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb557e16663ec35a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.353+0000 7f8cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0801d070 con 0x55a69e976620 2024-04-23T18:26:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f8d08005210 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0809c5b0 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f8d080a0960 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 10 client.16821 fetch_fsmap learned FSMap version 46 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a69e9903d0 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0809c5b0 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 46) ==== 8350+0+0 (secure 0 0 0) 0x7f8d08085af0 con 0x55a69e976620 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 10 client.16821 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:26:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 10 client.16821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 20 client.16821 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e991720 0x55a69e993b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 4 client.16821 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a69e8e9390 con 0x55a69e991720 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.354+0000 7f8d15414640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e991720 0x55a69e993b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.355+0000 7f8d15414640 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e991720 0x55a69e993b10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1caa7c3c06dafc66 server_cookie=fc04abf3bd5b652d in_seq=0 out_seq=0 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.355+0000 7f8cf57fa640 10 client.16821 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8cf57fa640 1 -- 172.21.15.26:0/4071803624 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a69e8e9390 con 0x55a69e991720 2024-04-23T18:26:34.359 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8cf57fa640 10 client.16821 handle_command_reply: tid=0 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 client.16821 shutdown 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 msgr2=0x55a69e981170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8d0809c540 tx=0x7f8d080870c0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 shutdown_connections 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e991720 0x55a69e993b10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e981800 0x55a69e97e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8cf57fa640 0 client.16821 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e976f40 0x55a69e97e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/4071803624 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e981170 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 >> 172.21.15.26:0/4071803624 conn(0x55a69e972b30 msgr2=0x55a69e978860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 shutdown_connections 2024-04-23T18:26:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/4071803624 wait complete. 2024-04-23T18:26:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.356+0000 7f8d1ab8f740 20 client.16821 trim_cache size 0 max 16384 2024-04-23T18:26:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.371+0000 7f8d1ab8f740 1 Processor -- start 2024-04-23T18:26:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 -- start start 2024-04-23T18:26:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976f40 0x55a69e97d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e9990e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101b2560 con 0x55a69e976620 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101afdd0 con 0x55a69e976f40 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101adc40 con 0x55a69e981800 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38644/0 (socket says 172.21.15.26:38644) 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 -- 172.21.15.26:0/2539574124 learned_addr learned my addr 172.21.15.26:0/2539574124 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 -- 172.21.15.26:0/2539574124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 msgr2=0x55a69e9990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e9990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 -- 172.21.15.26:0/2539574124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976f40 msgr2=0x55a69e97d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15414640 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976f40 0x55a69e97d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976f40 0x55a69e97d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 -- 172.21.15.26:0/2539574124 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69e97ed30 con 0x55a69e976620 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.372+0000 7f8d15c15640 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8d080ce360 tx=0x7f8d0809aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd0d17b05903564 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8cf77fe640 1 -- 172.21.15.26:0/2539574124 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0801d070 con 0x55a69e976620 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8cf77fe640 1 -- 172.21.15.26:0/2539574124 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0809c5b0 con 0x55a69e976620 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8cf77fe640 1 -- 172.21.15.26:0/2539574124 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08005210 con 0x55a69e976620 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/2539574124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 msgr2=0x55a69e98d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8d080ce360 tx=0x7f8d0809aac0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/2539574124 shutdown_connections 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976f40 0x55a69e97d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e9990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/2539574124 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976620 0x55a69e98d320 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/2539574124 >> 172.21.15.26:0/2539574124 conn(0x55a69e972b30 msgr2=0x55a69e971cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/2539574124 shutdown_connections 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.373+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/2539574124 wait complete. 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 -- start start 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 Processor -- start 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976620 0x55a69e989880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e986e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101ae290 con 0x55a69e976f40 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101af700 con 0x55a69e976620 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101ae960 con 0x55a69e981800 2024-04-23T18:26:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d1ab8f740 10 client.0 _ll_register_callbacks cb 0x7f8d080046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:38650/0 (socket says 172.21.15.26:38650) 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 -- 172.21.15.26:0/1389714073 learned_addr learned my addr 172.21.15.26:0/1389714073 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15c15640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976620 0x55a69e989880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d16416640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e986e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 -- 172.21.15.26:0/1389714073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 msgr2=0x55a69e986e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e986e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 -- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976620 msgr2=0x55a69e989880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.374+0000 7f8d15414640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976620 0x55a69e989880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d15414640 1 -- 172.21.15.26:0/1389714073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69e987610 con 0x55a69e976f40 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d16416640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e986e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d15414640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000c00 tx=0x7f8d04004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ae686b31da9b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0401f020 con 0x55a69e976f40 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f8d04003d20 con 0x55a69e976f40 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0809c5b0 con 0x55a69e976f40 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f8d0402a050 con 0x55a69e976f40 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 10 client.16827 fetch_fsmap learned FSMap version 46 2024-04-23T18:26:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a69e9f7660 con 0x55a69e976f40 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0809c5b0 con 0x55a69e976f40 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 46) ==== 8350+0+0 (secure 0 0 0) 0x7f8d0401b660 con 0x55a69e976f40 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 10 client.16827 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 10 client.16827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 20 client.16827 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e9f7e10 0x55a69e9fa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 4 client.16827 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.375+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a69e8e3200 con 0x55a69e9f7e10 2024-04-23T18:26:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.376+0000 7f8d15c15640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e9f7e10 0x55a69e9fa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.376+0000 7f8d15c15640 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e9f7e10 0x55a69e9fa230 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c814fceb1380baa7 server_cookie=1490c79407fc8f2 in_seq=0 out_seq=0 2024-04-23T18:26:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.376+0000 7f8cf57fa640 10 client.16827 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.401+0000 7f8cf57fa640 1 -- 172.21.15.26:0/1389714073 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a69e8e3200 con 0x55a69e9f7e10 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.401+0000 7f8cf57fa640 10 client.16827 handle_command_reply: tid=0 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.401+0000 7f8d1ab8f740 1 client.16827 shutdown 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 msgr2=0x55a69e989dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000c00 tx=0x7f8d04004410 comp rx=0 tx=0).stop 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 shutdown_connections 2024-04-23T18:26:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55a69e9f7e10 0x55a69e9fa230 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a69e981800 0x55a69e986e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55a69e976f40 0x55a69e989dc0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8cf57fa640 0 client.16827 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 --2- 172.21.15.26:0/1389714073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a69e976620 0x55a69e989880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 >> 172.21.15.26:0/1389714073 conn(0x55a69e972b30 msgr2=0x55a69e98c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 shutdown_connections 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 1 -- 172.21.15.26:0/1389714073 wait complete. 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d1ab8f740 20 client.16827 trim_cache size 0 max 16384 2024-04-23T18:26:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:26:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:26:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:26:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 msgr2=0x7f8cf80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.402+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 0x7f8cf80a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d00005e30 tx=0x7f8d000098c0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 msgr2=0x7f8d101a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800c970 tx=0x7f8d0800ce30 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 shutdown_connections 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f8cf80a3dc0 0x7f8cf80a6280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d10074ca0 0x7f8d101a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1010e790 0x7f8d1019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 --2- 172.21.15.26:0/3814305531 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8d10074380 0x7f8d101a05c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 >> 172.21.15.26:0/3814305531 conn(0x7f8d100fda20 msgr2=0x7f8d100fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 shutdown_connections 2024-04-23T18:26:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:34.403+0000 7f8d17e79640 1 -- 172.21.15.26:0/3814305531 wait complete. 2024-04-23T18:26:34.690 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.37 MB/sec execute 389 sec latency 7895.868 ms 2024-04-23T18:26:35.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:35 smithi110 ceph-mon[32585]: pgmap v292: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-23T18:26:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:35 smithi026 ceph-mon[29364]: pgmap v292: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-23T18:26:35.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:35 smithi131 ceph-mon[33041]: pgmap v292: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-23T18:26:35.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.35 MB/sec execute 390 sec latency 8895.942 ms 2024-04-23T18:26:36.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.34 MB/sec execute 391 sec latency 9896.050 ms 2024-04-23T18:26:37.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:37 smithi110 ceph-mon[32585]: pgmap v293: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-04-23T18:26:37.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:37 smithi026 ceph-mon[29364]: pgmap v293: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-04-23T18:26:37.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:37 smithi131 ceph-mon[33041]: pgmap v293: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-04-23T18:26:37.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.32 MB/sec execute 392 sec latency 10896.166 ms 2024-04-23T18:26:38.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.31 MB/sec execute 393 sec latency 11896.257 ms 2024-04-23T18:26:39.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:39 smithi110 ceph-mon[32585]: pgmap v294: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:39.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:39 smithi026 ceph-mon[29364]: pgmap v294: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:39.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:39 smithi131 ceph-mon[33041]: pgmap v294: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:39.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.30 MB/sec execute 394 sec latency 12896.355 ms 2024-04-23T18:26:40.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.28 MB/sec execute 395 sec latency 13896.483 ms 2024-04-23T18:26:41.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:41 smithi110 ceph-mon[32585]: pgmap v295: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:41.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:41 smithi026 ceph-mon[29364]: pgmap v295: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:41.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:41 smithi131 ceph-mon[33041]: pgmap v295: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-23T18:26:41.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.27 MB/sec execute 396 sec latency 14896.614 ms 2024-04-23T18:26:42.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.26 MB/sec execute 397 sec latency 15896.729 ms 2024-04-23T18:26:43.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:43 smithi110 ceph-mon[32585]: pgmap v296: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-04-23T18:26:43.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:43 smithi026 ceph-mon[29364]: pgmap v296: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-04-23T18:26:43.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:43 smithi131 ceph-mon[33041]: pgmap v296: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 78 op/s 2024-04-23T18:26:43.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.24 MB/sec execute 398 sec latency 16896.813 ms 2024-04-23T18:26:44.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.23 MB/sec execute 399 sec latency 17896.850 ms 2024-04-23T18:26:45.293 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:45 smithi131 ceph-mon[33041]: pgmap v297: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 526 KiB/s wr, 30 op/s 2024-04-23T18:26:45.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:45 smithi110 ceph-mon[32585]: pgmap v297: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 526 KiB/s wr, 30 op/s 2024-04-23T18:26:45.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:45 smithi026 ceph-mon[29364]: pgmap v297: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 526 KiB/s wr, 30 op/s 2024-04-23T18:26:45.691 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.22 MB/sec execute 400 sec latency 18896.975 ms 2024-04-23T18:26:46.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.20 MB/sec execute 401 sec latency 19897.059 ms 2024-04-23T18:26:47.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:47 smithi110 ceph-mon[32585]: pgmap v298: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 527 KiB/s wr, 33 op/s 2024-04-23T18:26:47.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:47 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:47 smithi026 ceph-mon[29364]: pgmap v298: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 527 KiB/s wr, 33 op/s 2024-04-23T18:26:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:47 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:47 smithi131 ceph-mon[33041]: pgmap v298: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 527 KiB/s wr, 33 op/s 2024-04-23T18:26:47.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:47 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 -- 172.21.15.26:0/2615287439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe014107280 msgr2=0x7fe01410ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 --2- 172.21.15.26:0/2615287439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe014107280 0x7fe01410ee50 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009ca0 tx=0x7fe00c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 -- 172.21.15.26:0/2615287439 shutdown_connections 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 --2- 172.21.15.26:0/2615287439 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe014107280 0x7fe01410ee50 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 --2- 172.21.15.26:0/2615287439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe014101070 0x7fe014106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 --2- 172.21.15.26:0/2615287439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0141006c0 0x7fe014100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 -- 172.21.15.26:0/2615287439 >> 172.21.15.26:0/2615287439 conn(0x7fe014077390 msgr2=0x7fe0140fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 -- 172.21.15.26:0/2615287439 shutdown_connections 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.502+0000 7fe01b18e640 1 -- 172.21.15.26:0/2615287439 wait complete. 2024-04-23T18:26:47.498 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 Processor -- start 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 -- start start 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0141006c0 0x7fe014101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe014107280 0x7fe014101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141121f0 con 0x7fe0141020b0 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe014112070 con 0x7fe014107280 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.503+0000 7fe01b18e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe014112370 con 0x7fe0141006c0 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33772/0 (socket says 172.21.15.26:33772) 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 -- 172.21.15.26:0/3054076275 learned_addr learned my addr 172.21.15.26:0/3054076275 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe018f2a640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0141006c0 0x7fe014101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe013fff640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe014107280 0x7fe014101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 -- 172.21.15.26:0/3054076275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0141006c0 msgr2=0x7fe014101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0141006c0 0x7fe014101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 -- 172.21.15.26:0/3054076275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe014107280 msgr2=0x7fe014101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe014107280 0x7fe014101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 -- 172.21.15.26:0/3054076275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0141a6f00 con 0x7fe0141020b0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01972b640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe00c005cc0 tx=0x7fe00c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe011ffb640 1 -- 172.21.15.26:0/3054076275 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c003bf0 con 0x7fe0141020b0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe011ffb640 1 -- 172.21.15.26:0/3054076275 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe00c003d90 con 0x7fe0141020b0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0141a7190 con 0x7fe0141020b0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.504+0000 7fe011ffb640 1 -- 172.21.15.26:0/3054076275 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c02fb20 con 0x7fe0141020b0 2024-04-23T18:26:47.500 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.505+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0141a76e0 con 0x7fe0141020b0 2024-04-23T18:26:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.507+0000 7fe011ffb640 1 -- 172.21.15.26:0/3054076275 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe00c004580 con 0x7fe0141020b0 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 Processor -- start 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 -- start start 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232721bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b232679970 con 0x55b232714620 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b2326164a0 con 0x55b23271f800 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01df7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b232675930 con 0x55b232714f40 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe011ffb640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 0x7fe0040a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe01972b640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 0x7fe0040a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33774/0 (socket says 172.21.15.26:33774) 2024-04-23T18:26:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 -- 172.21.15.26:0/1735143909 learned_addr learned my addr 172.21.15.26:0/1735143909 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe018f2a640 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe011ffb640 1 -- 172.21.15.26:0/3054076275 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fe00c0f1be0 con 0x7fe0141020b0 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232721bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 -- 172.21.15.26:0/1735143909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 msgr2=0x55b23271f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 -- 172.21.15.26:0/1735143909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 msgr2=0x55b232721bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.508+0000 7fe013fff640 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232721bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fe013fff640 1 -- 172.21.15.26:0/1735143909 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b23238c210 con 0x55b232714620 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fe01972b640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 0x7fe0040a62b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0bd930 tx=0x7fe00c08f040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fe013fff640 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x55b232711cc0 tx=0x7fe000004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e76e348e4d0b85 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fdffeffd640 1 -- 172.21.15.26:0/1735143909 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe000003e40 con 0x55b232714620 2024-04-23T18:26:47.509 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fdffeffd640 1 -- 172.21.15.26:0/1735143909 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0000054d0 con 0x55b232714620 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.509+0000 7fdffeffd640 1 -- 172.21.15.26:0/1735143909 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe000010460 con 0x55b232714620 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- 172.21.15.26:0/1735143909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 msgr2=0x55b232714a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x55b232711cc0 tx=0x7fe000004190 comp rx=0 tx=0).stop 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- 172.21.15.26:0/1735143909 shutdown_connections 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232721bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 --2- 172.21.15.26:0/1735143909 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b232714a00 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- 172.21.15.26:0/1735143909 >> 172.21.15.26:0/1735143909 conn(0x55b232710b30 msgr2=0x55b232710f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- 172.21.15.26:0/1735143909 shutdown_connections 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- 172.21.15.26:0/1735143909 wait complete. 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 -- start start 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.510+0000 7fe01df7e740 1 Processor -- start 2024-04-23T18:26:47.510 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b23271f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 0x55b23271c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55b2327223d0 con 0x55b23271f800 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b2326140b0 con 0x55b232714f40 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55b232722550 con 0x55b232714620 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe01df7e740 10 client.0 _ll_register_callbacks cb 0x55b232601d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33780/0 (socket says 172.21.15.26:33780) 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 -- 172.21.15.26:0/3961995394 learned_addr learned my addr 172.21.15.26:0/3961995394 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b23271f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe018f2a640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 0x55b23271c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 -- 172.21.15.26:0/3961995394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 msgr2=0x55b23271f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b23271f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 -- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 msgr2=0x55b23271c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 0x55b23271c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 -- 172.21.15.26:0/3961995394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b232458620 con 0x55b23271f800 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe018f2a640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 0x55b23271c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fe013fff640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe00000b850 tx=0x7fe000004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9038c524bd8575 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.511+0000 7fdff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000046c0 con 0x55b23271f800 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fe000004860 con 0x55b23271f800 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0000054d0 con 0x55b23271f800 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe000004a60 con 0x55b23271f800 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 10 client.16839 fetch_fsmap learned FSMap version 47 2024-04-23T18:26:47.512 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b23272e320 con 0x55b23271f800 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0000054d0 con 0x55b23271f800 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7fe00001f410 con 0x55b23271f800 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 10 client.16839 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 20 client.16839 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b23272f670 0x55b232731a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 4 client.16839 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.512+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b232687390 con 0x55b23272f670 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.513+0000 7fe018f2a640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b23272f670 0x55b232731a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.513+0000 7fdff3fff640 10 client.16839 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.513+0000 7fe018f2a640 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b23272f670 0x55b232731a60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82b28fa3275868c server_cookie=27205683db9ef7f1 in_seq=0 out_seq=0 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fdff3fff640 1 -- 172.21.15.26:0/3961995394 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b232687390 con 0x55b23272f670 2024-04-23T18:26:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fdff3fff640 10 client.16839 handle_command_reply: tid=0 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 client.16839 shutdown 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 msgr2=0x55b23271ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe00000b850 tx=0x7fe000004c30 comp rx=0 tx=0).stop 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 shutdown_connections 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b23272f670 0x55b232731a60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b232714f40 0x55b23271c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b23271f800 0x55b23271ca00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 --2- 172.21.15.26:0/3961995394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b23271f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fdff3fff640 0 client.16839 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 >> 172.21.15.26:0/3961995394 conn(0x55b232710b30 msgr2=0x55b232716860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 shutdown_connections 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 1 -- 172.21.15.26:0/3961995394 wait complete. 2024-04-23T18:26:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.514+0000 7fe01df7e740 20 client.16839 trim_cache size 0 max 16384 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 Processor -- start 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 -- start start 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b2327372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141b2610 con 0x55b232714620 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141afe50 con 0x55b23271f800 2024-04-23T18:26:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.529+0000 7fe01df7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141adcc0 con 0x55b232714f40 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33784/0 (socket says 172.21.15.26:33784) 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 -- 172.21.15.26:0/325008241 learned_addr learned my addr 172.21.15.26:0/325008241 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 -- 172.21.15.26:0/325008241 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 msgr2=0x55b23271b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 -- 172.21.15.26:0/325008241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 msgr2=0x55b2327372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b2327372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 -- 172.21.15.26:0/325008241 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b23271a9d0 con 0x55b232714620 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fe013fff640 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe00001f830 tx=0x7fe0000044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2db43b54719cbd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fdffe7fc640 1 -- 172.21.15.26:0/325008241 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00000b390 con 0x55b232714620 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fdffe7fc640 1 -- 172.21.15.26:0/325008241 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe00001fe50 con 0x55b232714620 2024-04-23T18:26:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.530+0000 7fdffe7fc640 1 -- 172.21.15.26:0/325008241 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00001faa0 con 0x55b232714620 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- 172.21.15.26:0/325008241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 msgr2=0x55b23272b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe00001f830 tx=0x7fe0000044f0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- 172.21.15.26:0/325008241 shutdown_connections 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b2327372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714f40 0x55b23271b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 --2- 172.21.15.26:0/325008241 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714620 0x55b23272b290 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- 172.21.15.26:0/325008241 >> 172.21.15.26:0/325008241 conn(0x55b232710b30 msgr2=0x55b23270fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- 172.21.15.26:0/325008241 shutdown_connections 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- 172.21.15.26:0/325008241 wait complete. 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 -- start start 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.531+0000 7fe01df7e740 1 Processor -- start 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b232727840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232725110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141ae310 con 0x55b232714f40 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141af780 con 0x55b23271f800 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe0141ae9e0 con 0x55b232714620 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe01df7e740 10 client.0 _ll_register_callbacks cb 0x7fe000003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33786/0 (socket says 172.21.15.26:33786) 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 -- 172.21.15.26:0/2315785167 learned_addr learned my addr 172.21.15.26:0/2315785167 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe013fff640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b232727840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe013fff640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232725110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 -- 172.21.15.26:0/2315785167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 msgr2=0x55b232727840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b232727840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 -- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 msgr2=0x55b232725110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232725110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 -- 172.21.15.26:0/2315785167 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b232725650 con 0x55b232714f40 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe013fff640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232725110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fe018f2a640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe00800adf0 tx=0x7fe008010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6d97dd206815a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.532+0000 7fdff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00801f540 con 0x55b232714f40 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fe00801f6e0 con 0x55b232714f40 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe00001fe50 con 0x55b232714f40 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe00801f8e0 con 0x55b232714f40 2024-04-23T18:26:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 10 client.16845 fetch_fsmap learned FSMap version 47 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b2327958c0 con 0x55b232714f40 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe00001fe50 con 0x55b232714f40 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7fe008031790 con 0x55b232714f40 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 10 client.16845 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 20 client.16845 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b232796050 0x55b232798470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 4 client.16845 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b232681200 con 0x55b232796050 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.533+0000 7fe013fff640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b232796050 0x55b232798470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.534+0000 7fe013fff640 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b232796050 0x55b232798470 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df9b6a272a425bc6 server_cookie=ad2424538b0c3b2 in_seq=0 out_seq=0 2024-04-23T18:26:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.534+0000 7fdff3fff640 10 client.16845 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fdff3fff640 1 -- 172.21.15.26:0/2315785167 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b232681200 con 0x55b232796050 2024-04-23T18:26:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fdff3fff640 10 client.16845 handle_command_reply: tid=0 2024-04-23T18:26:47.541 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 client.16845 shutdown 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 msgr2=0x55b232724bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe00800adf0 tx=0x7fe008010af0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 shutdown_connections 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55b232796050 0x55b232798470 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b23271f800 0x55b232725110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fdff3fff640 0 client.16845 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55b232714f40 0x55b232724bd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 --2- 172.21.15.26:0/2315785167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b232714620 0x55b232727840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 >> 172.21.15.26:0/2315785167 conn(0x55b232710b30 msgr2=0x55b23272a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 shutdown_connections 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 1 -- 172.21.15.26:0/2315785167 wait complete. 2024-04-23T18:26:47.542 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.548+0000 7fe01df7e740 20 client.16845 trim_cache size 0 max 16384 2024-04-23T18:26:47.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:26:47.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:26:47.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:26:47.543 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 msgr2=0x7fe0040a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 0x7fe0040a62b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0bd930 tx=0x7fe00c08f040 comp rx=0 tx=0).stop 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 msgr2=0x7fe0141a6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe00c005cc0 tx=0x7fe00c02f380 comp rx=0 tx=0).stop 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 shutdown_connections 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe0040a3df0 0x7fe0040a62b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe014107280 0x7fe014101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe0141020b0 0x7fe0141a6800 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 --2- 172.21.15.26:0/3054076275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0141006c0 0x7fe014101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:47.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 >> 172.21.15.26:0/3054076275 conn(0x7fe014077390 msgr2=0x7fe014105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:47.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 shutdown_connections 2024-04-23T18:26:47.545 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:26:47.549+0000 7fe01b18e640 1 -- 172.21.15.26:0/3054076275 wait complete. 2024-04-23T18:26:47.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.19 MB/sec execute 402 sec latency 20897.161 ms 2024-04-23T18:26:48.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.18 MB/sec execute 403 sec latency 21897.281 ms 2024-04-23T18:26:49.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:49 smithi110 ceph-mon[32585]: pgmap v299: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:49.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:49 smithi026 ceph-mon[29364]: pgmap v299: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:49.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:49 smithi131 ceph-mon[33041]: pgmap v299: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:49.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.17 MB/sec execute 404 sec latency 22897.372 ms 2024-04-23T18:26:50.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:26:50.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.597+0000 7fb63b20b640 1 -- 172.21.15.26:0/4151777530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 msgr2=0x7fb63410c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.597+0000 7fb63b20b640 1 --2- 172.21.15.26:0/4151777530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb63410c030 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb628009900 tx=0x7fb62802a480 comp rx=0 tx=0).stop 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 -- 172.21.15.26:0/4151777530 shutdown_connections 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- 172.21.15.26:0/4151777530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- 172.21.15.26:0/4151777530 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb63410c030 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- 172.21.15.26:0/4151777530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 0x7fb634103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 -- 172.21.15.26:0/4151777530 >> 172.21.15.26:0/4151777530 conn(0x7fb634077fb0 msgr2=0x7fb6340783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 -- 172.21.15.26:0/4151777530 shutdown_connections 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 -- 172.21.15.26:0/4151777530 wait complete. 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 Processor -- start 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 -- start start 2024-04-23T18:26:50.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 0x7fb634198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.598+0000 7fb63b20b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb63b20b640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb6341121b0 con 0x7fb634103b50 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb63b20b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb634112030 con 0x7fb63410c570 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb63b20b640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb634112330 con 0x7fb634103230 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39958/0 (socket says 172.21.15.26:39958) 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 -- 172.21.15.26:0/280075314 learned_addr learned my addr 172.21.15.26:0/280075314 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb638fa7640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 0x7fb634198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb6397a8640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 -- 172.21.15.26:0/280075314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 msgr2=0x7fb634198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 0x7fb634198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 -- 172.21.15.26:0/280075314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 msgr2=0x7fb63419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 -- 172.21.15.26:0/280075314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63419d600 con 0x7fb634103b50 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb6397a8640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb633fff640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb628006010 tx=0x7fb6280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62803f030 con 0x7fb634103b50 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb628003800 con 0x7fb634103b50 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63419d890 con 0x7fb634103b50 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.599+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb628035070 con 0x7fb634103b50 2024-04-23T18:26:50.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.600+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6341a5180 con 0x7fb634103b50 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.601+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6340691c0 con 0x7fb634103b50 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.602+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb628003cb0 con 0x7fb634103b50 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.603+0000 7fb631ffb640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 0x7fb61c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.603+0000 7fb638fa7640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 0x7fb61c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.603+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fb6280f1130 con 0x7fb634103b50 2024-04-23T18:26:50.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.603+0000 7fb638fa7640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 0x7fb61c0a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb62400abe0 tx=0x7fb62400a2d0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.608+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6280bd7d0 con 0x7fb634103b50 2024-04-23T18:26:50.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 540882 5.15 MB/sec execute 405 sec latency 23897.413 ms 2024-04-23T18:26:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.837+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6341a5380 con 0x7fb634103b50 2024-04-23T18:26:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.838+0000 7fb631ffb640 1 -- 172.21.15.26:0/280075314 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11889 (secure 0 0 0) 0x7fb6280bd0d0 con 0x7fb634103b50 2024-04-23T18:26:50.832 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:26:50.832 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:26:45.677081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24460,24464]},"id":1}]} 2024-04-23T18:26:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 47 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 msgr2=0x7fb61c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 0x7fb61c0a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb62400abe0 tx=0x7fb62400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 msgr2=0x7fb634198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb628006010 tx=0x7fb6280042f0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 shutdown_connections 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb61c0a3d20 0x7fb61c0a61e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb63410c570 0x7fb63419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb634103b50 0x7fb634198b50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 --2- 172.21.15.26:0/280075314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb634103230 0x7fb634198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 >> 172.21.15.26:0/280075314 conn(0x7fb634077fb0 msgr2=0x7fb6340fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 shutdown_connections 2024-04-23T18:26:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.840+0000 7fb63b20b640 1 -- 172.21.15.26:0/280075314 wait complete. 2024-04-23T18:26:50.845 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id 47cc512 --release --await' 2024-04-23T18:26:50.845 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id 47cc512 --release --await 2024-04-23T18:26:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 -- 172.21.15.26:0/1371024863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406b1f0 msgr2=0x7f2f74106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 --2- 172.21.15.26:0/1371024863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406b1f0 0x7f2f74106b90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009900 tx=0x7f2f6802a410 comp rx=0 tx=0).stop 2024-04-23T18:26:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 -- 172.21.15.26:0/1371024863 shutdown_connections 2024-04-23T18:26:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 --2- 172.21.15.26:0/1371024863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f74107270 0x7f2f7410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 --2- 172.21.15.26:0/1371024863 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406b1f0 0x7f2f74106b90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 --2- 172.21.15.26:0/1371024863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f7406a840 0x7f2f7406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 -- 172.21.15.26:0/1371024863 >> 172.21.15.26:0/1371024863 conn(0x7f2f74077eb0 msgr2=0x7f2f740782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 -- 172.21.15.26:0/1371024863 shutdown_connections 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.960+0000 7f2f7955c640 1 -- 172.21.15.26:0/1371024863 wait complete. 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 Processor -- start 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 -- start start 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 0x7f2f74198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f7419cfd0 0x7f2f7419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74112200 con 0x7f2f7406a840 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74112080 con 0x7f2f74107270 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f7955c640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74112380 con 0x7f2f7419cfd0 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39970/0 (socket says 172.21.15.26:39970) 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.961+0000 7f2f72ffd640 1 -- 172.21.15.26:0/118710823 learned_addr learned my addr 172.21.15.26:0/118710823 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f727fc640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 0x7f2f74198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 -- 172.21.15.26:0/118710823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f7419cfd0 msgr2=0x7f2f7419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f737fe640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f7419cfd0 0x7f2f7419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f7419cfd0 0x7f2f7419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 -- 172.21.15.26:0/118710823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 msgr2=0x7f2f74198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 0x7f2f74198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 -- 172.21.15.26:0/118710823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7419dac0 con 0x7f2f7406a840 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f727fc640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 0x7f2f74198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400d950 tx=0x7f2f6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f5bfff640 1 -- 172.21.15.26:0/118710823 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64017070 con 0x7f2f7406a840 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7419ddb0 con 0x7f2f7406a840 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f5bfff640 1 -- 172.21.15.26:0/118710823 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f64009c30 con 0x7f2f7406a840 2024-04-23T18:26:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f5bfff640 1 -- 172.21.15.26:0/118710823 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400f620 con 0x7f2f7406a840 2024-04-23T18:26:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.962+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f741a5570 con 0x7f2f7406a840 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.964+0000 7f2f5bfff640 1 -- 172.21.15.26:0/118710823 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2f64010460 con 0x7f2f7406a840 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.964+0000 7f2f7c270740 1 Processor -- start 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 -- start start 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7490 0x55643dca7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcb1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55643dc0c6d0 con 0x55643dcb2480 2024-04-23T18:26:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55643dba91a0 con 0x55643dca7db0 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f7c270740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55643dc08690 con 0x55643dca7490 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f5bfff640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 0x7f2f5c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39986/0 (socket says 172.21.15.26:39986) 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 -- 172.21.15.26:0/1824929902 learned_addr learned my addr 172.21.15.26:0/1824929902 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f737fe640 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7490 0x55643dca7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f737fe640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 0x7f2f5c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f5bfff640 1 -- 172.21.15.26:0/118710823 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f2f640d04f0 con 0x7f2f7406a840 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcb1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 -- 172.21.15.26:0/1824929902 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7490 msgr2=0x55643dca7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7490 0x55643dca7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 -- 172.21.15.26:0/1824929902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 msgr2=0x55643dcb1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcb1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 -- 172.21.15.26:0/1824929902 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643dbb35b0 con 0x55643dcb2480 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f737fe640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 0x7f2f5c0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0092f0 tx=0x7f2f6c004a40 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.965+0000 7f2f727fc640 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2f7406a420 tx=0x7f2f680041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133767aae810cdcb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f597fa640 1 -- 172.21.15.26:0/1824929902 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68035030 con 0x55643dcb2480 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f597fa640 1 -- 172.21.15.26:0/1824929902 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f680054f0 con 0x55643dcb2480 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f597fa640 1 -- 172.21.15.26:0/1824929902 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f680035f0 con 0x55643dcb2480 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 -- 172.21.15.26:0/1824929902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 msgr2=0x55643dcb4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2f7406a420 tx=0x7f2f680041f0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 -- 172.21.15.26:0/1824929902 shutdown_connections 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcb1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb4870 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 --2- 172.21.15.26:0/1824929902 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7490 0x55643dca7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 -- 172.21.15.26:0/1824929902 >> 172.21.15.26:0/1824929902 conn(0x55643dca37b0 msgr2=0x55643dca3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 -- 172.21.15.26:0/1824929902 shutdown_connections 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.966+0000 7f2f7c270740 1 -- 172.21.15.26:0/1824929902 wait complete. 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 -- start start 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 Processor -- start 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcaf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 0x55643dcaf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55643dcb52b0 con 0x55643dca7490 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55643dba6b20 con 0x55643dcb2480 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55643dcb5430 con 0x55643dca7db0 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f7c270740 10 client.0 _ll_register_callbacks cb 0x55643dba6530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:39998/0 (socket says 172.21.15.26:39998) 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 -- 172.21.15.26:0/156443690 learned_addr learned my addr 172.21.15.26:0/156443690 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f737fe640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 0x55643dcaf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f727fc640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcaf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 -- 172.21.15.26:0/156443690 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 msgr2=0x55643dcaf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcaf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.967+0000 7f2f72ffd640 1 -- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 msgr2=0x55643dcaf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 0x55643dcaf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f72ffd640 1 -- 172.21.15.26:0/156443690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643d91e920 con 0x55643dca7490 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f737fe640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 0x55643dcaf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2f6409c190 tx=0x7f2f6409cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80b55df003ab96f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64020040 con 0x55643dca7490 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f2f640d0070 con 0x55643dca7490 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f680054f0 con 0x55643dca7490 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2f6409c490 con 0x55643dca7490 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 10 client.16863 fetch_fsmap learned FSMap version 47 2024-04-23T18:26:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55643d9ead30 con 0x55643dca7490 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f680054f0 con 0x55643dca7490 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f2f640bb2b0 con 0x55643dca7490 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 10 client.16863 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 10 client.16863 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 20 client.16863 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dcc25b0 0x55643dcc4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 4 client.16863 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.968+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55643dbca500 con 0x55643dcc25b0 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.969+0000 7f2f727fc640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dcc25b0 0x55643dcc4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.969+0000 7f2f727fc640 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dcc25b0 0x55643dcc4a00 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dbbbb47e91abd6f server_cookie=689c97d1def60c41 in_seq=0 out_seq=0 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.969+0000 7f2f56ff5640 10 client.16863 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f56ff5640 1 -- 172.21.15.26:0/156443690 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55643dbca500 con 0x55643dcc25b0 2024-04-23T18:26:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f56ff5640 10 client.16863 handle_command_reply: tid=0 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 client.16863 shutdown 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 msgr2=0x55643dcb1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2f6409c190 tx=0x7f2f6409cb90 comp rx=0 tx=0).stop 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 shutdown_connections 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dcc25b0 0x55643dcc4a00 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dcb2480 0x55643dcaf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f56ff5640 0 client.16863 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcaf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 --2- 172.21.15.26:0/156443690 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dcb1eb0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 >> 172.21.15.26:0/156443690 conn(0x55643dca37b0 msgr2=0x55643dca95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 shutdown_connections 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 1 -- 172.21.15.26:0/156443690 wait complete. 2024-04-23T18:26:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.970+0000 7f2f7c270740 20 client.16863 trim_cache size 0 max 16384 2024-04-23T18:26:50.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 Processor -- start 2024-04-23T18:26:50.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 -- start start 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcbe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 0x55643dcb19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741b03a0 con 0x55643dca7490 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741adbe0 con 0x55643dca7db0 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.987+0000 7f2f7c270740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741aba50 con 0x55643dcb2480 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40004/0 (socket says 172.21.15.26:40004) 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 -- 172.21.15.26:0/21052399 learned_addr learned my addr 172.21.15.26:0/21052399 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f737fe640 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 0x55643dcb19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 -- 172.21.15.26:0/21052399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 msgr2=0x55643dcb19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 0x55643dcb19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 -- 172.21.15.26:0/21052399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 msgr2=0x55643dcbe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcbe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 -- 172.21.15.26:0/21052399 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643dcae150 con 0x55643dca7490 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f737fe640 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 0x55643dcb19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2f640ce240 tx=0x7f2f64009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9cbd357824d9e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f58ff9640 1 -- 172.21.15.26:0/21052399 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64020050 con 0x55643dca7490 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f58ff9640 1 -- 172.21.15.26:0/21052399 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f640a0840 con 0x55643dca7490 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.988+0000 7f2f58ff9640 1 -- 172.21.15.26:0/21052399 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64010a50 con 0x55643dca7490 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- 172.21.15.26:0/21052399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 msgr2=0x55643dca9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2f640ce240 tx=0x7f2f64009740 comp rx=0 tx=0).stop 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- 172.21.15.26:0/21052399 shutdown_connections 2024-04-23T18:26:50.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7db0 0x55643dcbe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dcb2480 0x55643dcb19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- 172.21.15.26:0/21052399 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dca7490 0x55643dca9c00 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- 172.21.15.26:0/21052399 >> 172.21.15.26:0/21052399 conn(0x55643dca37b0 msgr2=0x55643dca94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- 172.21.15.26:0/21052399 shutdown_connections 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- 172.21.15.26:0/21052399 wait complete. 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- start start 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 Processor -- start 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7490 0x55643dcba690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcbabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741ac0a0 con 0x55643dcb2480 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741ad510 con 0x55643dca7490 2024-04-23T18:26:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741ac770 con 0x55643dca7db0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f7c270740 10 client.0 _ll_register_callbacks cb 0x7f2f6409bfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f737fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f737fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:40014/0 (socket says 172.21.15.26:40014) 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f737fe640 1 -- 172.21.15.26:0/2091151019 learned_addr learned my addr 172.21.15.26:0/2091151019 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.989+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7490 0x55643dcba690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f727fc640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcbabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 -- 172.21.15.26:0/2091151019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 msgr2=0x55643dcbabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcbabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 -- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7490 msgr2=0x55643dcba690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7490 0x55643dcba690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 -- 172.21.15.26:0/2091151019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643dcb5e40 con 0x55643dcb2480 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f737fe640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c005fd0 tx=0x7f2f6c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e7602aef198041 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c004b20 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f2f6c004cc0 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f640a0840 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2f6c01e640 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f7c270740 10 client.16869 fetch_fsmap learned FSMap version 47 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55643dd29480 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.990+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f640a0840 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 47) ==== 8354+0+0 (secure 0 0 0) 0x7f2f6c00f040 con 0x55643dcb2480 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 10 client.16869 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 10 client.16869 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 20 client.16869 populate_metadata read hostname 'smithi026' 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dd29cb0 0x55643dd2c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:50.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 4 client.16869 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "47cc512", "release": true, "await": true}] 2024-04-23T18:26:50.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "set_id": "47cc512", "release": true, "await": true}) -- 0x55643dc1a0f0 con 0x55643dd29cb0 2024-04-23T18:26:50.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dd29cb0 0x55643dd2c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:50.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f56ff5640 10 client.16869 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:50.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.991+0000 7f2f72ffd640 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dd29cb0 0x55643dd2c0d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8371325f6b817ed4 server_cookie=c97653f8492f8cd3 in_seq=0 out_seq=0 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f56ff5640 1 -- 172.21.15.26:0/2091151019 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55643dc1a0f0 con 0x55643dd29cb0 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f56ff5640 10 client.16869 handle_command_reply: tid=0 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 client.16869 shutdown 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 msgr2=0x55643dcb5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c005fd0 tx=0x7f2f6c004ab0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 shutdown_connections 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55643dd29cb0 0x55643dd2c0d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55643dcb2480 0x55643dcb5640 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55643dca7db0 0x55643dcbabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 --2- 172.21.15.26:0/2091151019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55643dca7490 0x55643dcba690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f56ff5640 0 client.16869 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.995+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 >> 172.21.15.26:0/2091151019 conn(0x55643dca37b0 msgr2=0x55643dca8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 shutdown_connections 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7c270740 1 -- 172.21.15.26:0/2091151019 wait complete. 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7c270740 20 client.16869 trim_cache size 0 max 16384 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 47, 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 44, 2024-04-23T18:26:50.993 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "47cc512": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "version": 44, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age_ref": 0.0, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "timeout": 60.0, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "expiration": 83.7, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "members": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "file:/": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "excluded": false, 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "state": { 2024-04-23T18:26:50.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RELEASED", 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:26:50.995 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 msgr2=0x7f2f5c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 0x7f2f5c0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0092f0 tx=0x7f2f6c004a40 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 msgr2=0x7f2f74198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400d950 tx=0x7f2f6400de10 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 shutdown_connections 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f5c0a3d20 0x7f2f5c0a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f74107270 0x7f2f74198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f7419cfd0 0x7f2f7419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 --2- 172.21.15.26:0/118710823 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f7406a840 0x7f2f74198760 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 >> 172.21.15.26:0/118710823 conn(0x7f2f74077eb0 msgr2=0x7f2f74100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 shutdown_connections 2024-04-23T18:26:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:26:50.996+0000 7f2f7955c640 1 -- 172.21.15.26:0/118710823 wait complete. 2024-04-23T18:26:51.001 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '47cc512', seconds elapsed: 24.8 2024-04-23T18:26:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:51 smithi026 ceph-mon[29364]: pgmap v300: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:51 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/280075314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:51 smithi110 ceph-mon[32585]: pgmap v300: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:51.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:51 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/280075314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:51 smithi131 ceph-mon[33041]: pgmap v300: 97 pgs: 97 active+clean; 356 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 10 op/s 2024-04-23T18:26:51.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:51 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/280075314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:26:51.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 542375 5.16 MB/sec execute 406 sec latency 24198.748 ms 2024-04-23T18:26:52.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 542856 5.15 MB/sec execute 407 sec latency 771.890 ms 2024-04-23T18:26:53.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:53 smithi110 ceph-mon[32585]: pgmap v301: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 43 KiB/s wr, 17 op/s 2024-04-23T18:26:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:53 smithi026 ceph-mon[29364]: pgmap v301: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 43 KiB/s wr, 17 op/s 2024-04-23T18:26:53.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:53 smithi131 ceph-mon[33041]: pgmap v301: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 43 KiB/s wr, 17 op/s 2024-04-23T18:26:53.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 542856 5.14 MB/sec execute 408 sec latency 1771.991 ms 2024-04-23T18:26:54.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 543186 5.13 MB/sec execute 409 sec latency 2500.271 ms 2024-04-23T18:26:55.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:55 smithi110 ceph-mon[32585]: pgmap v302: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:26:55.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:55 smithi026 ceph-mon[29364]: pgmap v302: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:26:55.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:55 smithi131 ceph-mon[33041]: pgmap v302: 97 pgs: 97 active+clean; 352 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:26:55.693 INFO:tasks.workunit.client.0.smithi026.stdout: 1 543186 5.12 MB/sec execute 410 sec latency 1101.297 ms 2024-04-23T18:26:56.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 543520 5.12 MB/sec execute 411 sec latency 1948.869 ms 2024-04-23T18:26:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:57 smithi110 ceph-mon[32585]: pgmap v303: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 224 KiB/s wr, 34 op/s 2024-04-23T18:26:57.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:57 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:57.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:57 smithi026 ceph-mon[29364]: pgmap v303: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 224 KiB/s wr, 34 op/s 2024-04-23T18:26:57.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:57 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:57 smithi131 ceph-mon[33041]: pgmap v303: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 224 KiB/s wr, 34 op/s 2024-04-23T18:26:57.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:57 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:57.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 544231 5.11 MB/sec execute 412 sec latency 24.349 ms 2024-04-23T18:26:58.693 INFO:tasks.workunit.client.0.smithi026.stdout: 1 546079 5.12 MB/sec execute 413 sec latency 13.310 ms 2024-04-23T18:26:59.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:26:59 smithi110 ceph-mon[32585]: pgmap v304: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:26:59.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:26:59 smithi026 ceph-mon[29364]: pgmap v304: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:26:59.490 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:26:59 smithi131 ceph-mon[33041]: pgmap v304: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:26:59.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 547484 5.13 MB/sec execute 414 sec latency 11.937 ms 2024-04-23T18:27:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 -- 172.21.15.26:0/3310336285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 msgr2=0x7fbce01025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 --2- 172.21.15.26:0/3310336285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce01025b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009900 tx=0x7fbcdc02a410 comp rx=0 tx=0).stop 2024-04-23T18:27:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 -- 172.21.15.26:0/3310336285 shutdown_connections 2024-04-23T18:27:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 --2- 172.21.15.26:0/3310336285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce0102b80 0x7fbce0106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 --2- 172.21.15.26:0/3310336285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce01072a0 0x7fbce010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 --2- 172.21.15.26:0/3310336285 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce01025b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 -- 172.21.15.26:0/3310336285 >> 172.21.15.26:0/3310336285 conn(0x7fbce00793e0 msgr2=0x7fbce00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 -- 172.21.15.26:0/3310336285 shutdown_connections 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.638+0000 7fbce73bb640 1 -- 172.21.15.26:0/3310336285 wait complete. 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 Processor -- start 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 -- start start 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 0x7fbce0198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce019ced0 0x7fbce019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0112210 con 0x7fbce01021d0 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0112090 con 0x7fbce01072a0 2024-04-23T18:27:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce73bb640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0112390 con 0x7fbce019ced0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41470/0 (socket says 172.21.15.26:41470) 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 -- 172.21.15.26:0/4058047883 learned_addr learned my addr 172.21.15.26:0/4058047883 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5958640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce019ced0 0x7fbce019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce4956640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 0x7fbce0198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 -- 172.21.15.26:0/4058047883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce019ced0 msgr2=0x7fbce019d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce019ced0 0x7fbce019d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 -- 172.21.15.26:0/4058047883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 msgr2=0x7fbce0198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 0x7fbce0198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce5157640 1 -- 172.21.15.26:0/4058047883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce019d9c0 con 0x7fbce01021d0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.639+0000 7fbce4956640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 0x7fbce0198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbce5157640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc005940 tx=0x7fbcdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbcd67fc640 1 -- 172.21.15.26:0/4058047883 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc03f030 con 0x7fbce01021d0 2024-04-23T18:27:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbcd67fc640 1 -- 172.21.15.26:0/4058047883 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcdc003aa0 con 0x7fbce01021d0 2024-04-23T18:27:00.637 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce019dc50 con 0x7fbce01021d0 2024-04-23T18:27:00.637 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbcd67fc640 1 -- 172.21.15.26:0/4058047883 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc035070 con 0x7fbce01021d0 2024-04-23T18:27:00.637 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.640+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01a5410 con 0x7fbce01021d0 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.642+0000 7fbcd67fc640 1 -- 172.21.15.26:0/4058047883 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbcdc042030 con 0x7fbce01021d0 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 Processor -- start 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 -- start start 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a50ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a519bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a471970 con 0x55dd5a50cf40 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a40e4a0 con 0x55dd5a517800 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a46d930 con 0x55dd5a50c620 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcd67fc640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 0x7fbccc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41482/0 (socket says 172.21.15.26:41482) 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 -- 172.21.15.26:0/3845577263 learned_addr learned my addr 172.21.15.26:0/3845577263 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5958640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 0x7fbccc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5958640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a50ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce4956640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a519bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbcd67fc640 1 -- 172.21.15.26:0/4058047883 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fbcdc0f0db0 con 0x7fbce01021d0 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 -- 172.21.15.26:0/3845577263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 msgr2=0x55dd5a50ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a50ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 -- 172.21.15.26:0/3845577263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 msgr2=0x55dd5a519bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a519bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce5157640 1 -- 172.21.15.26:0/3845577263 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5a184210 con 0x55dd5a50cf40 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.643+0000 7fbce4956640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a519bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbce5958640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 0x7fbccc0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009810 tx=0x7fbcd8009b00 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbce5157640 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0ee470 tx=0x7fbcdc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd8c0db5f59dd55 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbccb7fe640 1 -- 172.21.15.26:0/3845577263 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc049070 con 0x55dd5a50cf40 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbccb7fe640 1 -- 172.21.15.26:0/3845577263 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcdc0c1100 con 0x55dd5a50cf40 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbccb7fe640 1 -- 172.21.15.26:0/3845577263 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0f30d0 con 0x55dd5a50cf40 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3845577263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 msgr2=0x55dd5a5172c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.644+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0ee470 tx=0x7fbcdc003f10 comp rx=0 tx=0).stop 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3845577263 shutdown_connections 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a519bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a5172c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3845577263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a50ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3845577263 >> 172.21.15.26:0/3845577263 conn(0x55dd5a508b30 msgr2=0x55dd5a508f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3845577263 shutdown_connections 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3845577263 wait complete. 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- start start 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 Processor -- start 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a514460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a5149c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a471970 con 0x55dd5a50c620 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a40c0b0 con 0x55dd5a517800 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a51a1f0 con 0x55dd5a50cf40 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbcea1ab740 10 client.0 _ll_register_callbacks cb 0x55dd5a3f9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.645+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41490/0 (socket says 172.21.15.26:41490) 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 -- 172.21.15.26:0/3319519619 learned_addr learned my addr 172.21.15.26:0/3319519619 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce4956640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a514460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5958640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a5149c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 -- 172.21.15.26:0/3319519619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 msgr2=0x55dd5a514460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a514460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 -- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 msgr2=0x55dd5a5149c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a5149c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 -- 172.21.15.26:0/3319519619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5a250620 con 0x55dd5a50c620 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce4956640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a514460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5958640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a5149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbce5157640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0bcce0 tx=0x7fbcdc003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec5fc4db55739c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc049040 con 0x55dd5a50c620 2024-04-23T18:27:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fbcdc0bcfb0 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcdc0c1100 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fbcdc0d6270 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcea1ab740 10 client.16881 fetch_fsmap learned FSMap version 48 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dd5a526010 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.646+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcdc0c1100 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7fbcdc0dd110 con 0x55dd5a50c620 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 10 client.16881 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 10 client.16881 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 20 client.16881 populate_metadata read hostname 'smithi026' 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a5273b0 0x55dd5a5297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 4 client.16881 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd5a47f390 con 0x55dd5a5273b0 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbce4956640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a5273b0 0x55dd5a5297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbcc8ff9640 10 client.16881 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.647+0000 7fbce4956640 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a5273b0 0x55dd5a5297e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd90d60d297335e server_cookie=47d003aa34025d66 in_seq=0 out_seq=0 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/3319519619 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd5a47f390 con 0x55dd5a5273b0 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcc8ff9640 10 client.16881 handle_command_reply: tid=0 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 client.16881 shutdown 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 msgr2=0x55dd5a517170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0bcce0 tx=0x7fbcdc003e80 comp rx=0 tx=0).stop 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 shutdown_connections 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a5273b0 0x55dd5a5297e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a5149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a514460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcc8ff9640 0 client.16881 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/3319519619 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50c620 0x55dd5a517170 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 >> 172.21.15.26:0/3319519619 conn(0x55dd5a508b30 msgr2=0x55dd5a50e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 shutdown_connections 2024-04-23T18:27:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 1 -- 172.21.15.26:0/3319519619 wait complete. 2024-04-23T18:27:00.650 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.648+0000 7fbcea1ab740 20 client.16881 trim_cache size 0 max 16384 2024-04-23T18:27:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 Processor -- start 2024-04-23T18:27:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 -- start start 2024-04-23T18:27:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a522f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a52f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01b0430 con 0x55dd5a50cf40 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01adc40 con 0x55dd5a517800 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01abab0 con 0x55dd5a50c620 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbce4956640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbce4956640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41506/0 (socket says 172.21.15.26:41506) 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbce4956640 1 -- 172.21.15.26:0/2009122144 learned_addr learned my addr 172.21.15.26:0/2009122144 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbce5157640 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a522f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.663+0000 7fbce4956640 1 -- 172.21.15.26:0/2009122144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 msgr2=0x55dd5a522f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce4956640 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a522f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce4956640 1 -- 172.21.15.26:0/2009122144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 msgr2=0x55dd5a52f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.662 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce4956640 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a52f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce4956640 1 -- 172.21.15.26:0/2009122144 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5a513410 con 0x55dd5a50cf40 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce5157640 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a522f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbce4956640 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000dd30 tx=0x7fbcd000d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94dd801e60c41d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbccaffd640 1 -- 172.21.15.26:0/2009122144 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0012aa0 con 0x55dd5a50cf40 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbccaffd640 1 -- 172.21.15.26:0/2009122144 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd00054d0 con 0x55dd5a50cf40 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbccaffd640 1 -- 172.21.15.26:0/2009122144 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0019840 con 0x55dd5a50cf40 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2009122144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 msgr2=0x55dd5a507b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000dd30 tx=0x7fbcd000d770 comp rx=0 tx=0).stop 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2009122144 shutdown_connections 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a517800 0x55dd5a52f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a50cf40 0x55dd5a507b30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.664+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2009122144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50c620 0x55dd5a522f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2009122144 >> 172.21.15.26:0/2009122144 conn(0x55dd5a508b30 msgr2=0x55dd5a507420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2009122144 shutdown_connections 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2009122144 wait complete. 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- start start 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 Processor -- start 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a50c620 0x55dd5a51cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a51d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01ac100 con 0x55dd5a517800 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01ad570 con 0x55dd5a50c620 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01ac7d0 con 0x55dd5a50cf40 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbcea1ab740 10 client.0 _ll_register_callbacks cb 0x7fbcdc0f10b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbce5958640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbce5958640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41520/0 (socket says 172.21.15.26:41520) 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbce5958640 1 -- 172.21.15.26:0/2937374169 learned_addr learned my addr 172.21.15.26:0/2937374169 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.665+0000 7fbce4956640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a51d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5157640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a50c620 0x55dd5a51cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 -- 172.21.15.26:0/2937374169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 msgr2=0x55dd5a51d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a51d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 -- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a50c620 msgr2=0x55dd5a51cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a50c620 0x55dd5a51cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 -- 172.21.15.26:0/2937374169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5a50cd70 con 0x55dd5a517800 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbce5958640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8011f60 tx=0x7fbcd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885750e597a52a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8025070 con 0x55dd5a517800 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fbcd80047c0 con 0x55dd5a517800 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcd00054d0 con 0x55dd5a517800 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fbcd8031050 con 0x55dd5a517800 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcea1ab740 10 client.16887 fetch_fsmap learned FSMap version 48 2024-04-23T18:27:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dd5a58da40 con 0x55dd5a517800 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.666+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd00054d0 con 0x55dd5a517800 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7fbcd800f4f0 con 0x55dd5a517800 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 10 client.16887 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 20 client.16887 populate_metadata read hostname 'smithi026' 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a58e220 0x55dd5a590640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 4 client.16887 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd5a479200 con 0x55dd5a58e220 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbce5157640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a58e220 0x55dd5a590640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbcc8ff9640 10 client.16887 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.667+0000 7fbce5157640 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a58e220 0x55dd5a590640 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa1beb60f58768b8 server_cookie=7801bbe893aa0e0c in_seq=0 out_seq=0 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcc8ff9640 1 -- 172.21.15.26:0/2937374169 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd5a479200 con 0x55dd5a58e220 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcc8ff9640 10 client.16887 handle_command_reply: tid=0 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 client.16887 shutdown 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 msgr2=0x55dd5a51d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8011f60 tx=0x7fbcd8004a40 comp rx=0 tx=0).stop 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 shutdown_connections 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x55dd5a58e220 0x55dd5a590640 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x55dd5a517800 0x55dd5a51d9a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcc8ff9640 0 client.16887 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd5a50cf40 0x55dd5a51d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 --2- 172.21.15.26:0/2937374169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dd5a50c620 0x55dd5a51cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 >> 172.21.15.26:0/2937374169 conn(0x55dd5a508b30 msgr2=0x55dd5a507cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 shutdown_connections 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 1 -- 172.21.15.26:0/2937374169 wait complete. 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbcea1ab740 20 client.16887 trim_cache size 0 max 16384 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:{ 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "message": "", 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout: "return_code": 0 2024-04-23T18:27:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stdout:} 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 msgr2=0x7fbccc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 0x7fbccc0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009810 tx=0x7fbcd8009b00 comp rx=0 tx=0).stop 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 msgr2=0x7fbce0198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc005940 tx=0x7fbcdc004270 comp rx=0 tx=0).stop 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 shutdown_connections 2024-04-23T18:27:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fbccc0a3dc0 0x7fbccc0a6280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbce01072a0 0x7fbce0198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbce019ced0 0x7fbce019d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 --2- 172.21.15.26:0/4058047883 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fbce01021d0 0x7fbce0198610 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 >> 172.21.15.26:0/4058047883 conn(0x7fbce00793e0 msgr2=0x7fbce00ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 shutdown_connections 2024-04-23T18:27:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi026.front.sepia.ceph.com.smithi026.stderr:2024-04-23T18:27:00.691+0000 7fbce73bb640 1 -- 172.21.15.26:0/4058047883 wait complete. 2024-04-23T18:27:00.692 INFO:tasks.workunit.client.0.smithi026.stdout: 1 548302 5.12 MB/sec execute 415 sec latency 392.385 ms 2024-04-23T18:27:01.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:01 smithi110 ceph-mon[32585]: pgmap v305: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:27:01.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:01 smithi026 ceph-mon[29364]: pgmap v305: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:27:01.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:01 smithi131 ceph-mon[33041]: pgmap v305: 97 pgs: 97 active+clean; 354 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 223 KiB/s wr, 31 op/s 2024-04-23T18:27:01.693 INFO:tasks.workunit.client.0.smithi026.stdout: 1 548302 5.11 MB/sec execute 416 sec latency 1392.471 ms 2024-04-23T18:27:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:27:02.166 INFO:tasks.workunit.client.0.smithi026.stdout:[548342] write failed on handle 12128 (Resource temporarily unavailable) 2024-04-23T18:27:02.166 INFO:tasks.workunit.client.0.smithi026.stdout:Child failed with status 1 2024-04-23T18:27:02.166 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-23T18:27:02.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T18:27:02.528 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T18:27:02.771 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=76ef22ae47c54a3e9262fe71a7b3b927 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T18:27:02.774 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-23T18:27:02.784 INFO:tasks.quiescer:joining Quiescers 2024-04-23T18:27:02.784 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-23T18:27:02.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:02.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 -- 172.21.15.26:0/1436241365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c60102240 msgr2=0x7f3c60102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 --2- 172.21.15.26:0/1436241365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c60102240 0x7f3c60102620 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009900 tx=0x7f3c5002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 -- 172.21.15.26:0/1436241365 shutdown_connections 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 --2- 172.21.15.26:0/1436241365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60107280 0x7f3c6010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 --2- 172.21.15.26:0/1436241365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60102bf0 0x7f3c60106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 --2- 172.21.15.26:0/1436241365 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c60102240 0x7f3c60102620 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 -- 172.21.15.26:0/1436241365 >> 172.21.15.26:0/1436241365 conn(0x7f3c600790e0 msgr2=0x7f3c600794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 -- 172.21.15.26:0/1436241365 shutdown_connections 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.898+0000 7f3c66719640 1 -- 172.21.15.26:0/1436241365 wait complete. 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 Processor -- start 2024-04-23T18:27:02.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 -- start start 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60102240 0x7f3c6019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60107280 0x7f3c6019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f3c60112240 con 0x7f3c601985e0 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c601120c0 con 0x7f3c60102240 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.900+0000 7f3c66719640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3c601123c0 con 0x7f3c60107280 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41538/0 (socket says 172.21.15.26:41538) 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 -- 172.21.15.26:0/2625510327 learned_addr learned my addr 172.21.15.26:0/2625510327 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c5ffff640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60102240 0x7f3c6019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c5f7fe640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60107280 0x7f3c6019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 -- 172.21.15.26:0/2625510327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60107280 msgr2=0x7f3c6019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60107280 0x7f3c6019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 -- 172.21.15.26:0/2625510327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60102240 msgr2=0x7f3c6019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60102240 0x7f3c6019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 -- 172.21.15.26:0/2625510327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c60199100 con 0x7f3c601985e0 2024-04-23T18:27:02.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c64cb6640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800eaa0 tx=0x7f3c5800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:02.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800cdb0 con 0x7f3c601985e0 2024-04-23T18:27:02.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c58005020 con 0x7f3c601985e0 2024-04-23T18:27:02.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c601993f0 con 0x7f3c601985e0 2024-04-23T18:27:02.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580116d0 con 0x7f3c601985e0 2024-04-23T18:27:02.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.901+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c601a55f0 con 0x7f3c601985e0 2024-04-23T18:27:02.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.902+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c600691c0 con 0x7f3c601985e0 2024-04-23T18:27:02.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.903+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3c580051d0 con 0x7f3c601985e0 2024-04-23T18:27:02.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.903+0000 7f3c5d7fa640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 0x7f3c480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:02.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.903+0000 7f3c5ffff640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 0x7f3c480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:02.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.903+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3c580d0a70 con 0x7f3c601985e0 2024-04-23T18:27:02.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.905+0000 7f3c5ffff640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 0x7f3c480a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3c50031440 tx=0x7f3c50005b50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:02.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:02.909+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c58016030 con 0x7f3c601985e0 2024-04-23T18:27:03.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.134+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c60199b10 con 0x7f3c601985e0 2024-04-23T18:27:03.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.135+0000 7f3c5d7fa640 1 -- 172.21.15.26:0/2625510327 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11890 (secure 0 0 0) 0x7f3c60199b10 con 0x7f3c601985e0 2024-04-23T18:27:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:27:02.123441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:27:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 49 2024-04-23T18:27:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 msgr2=0x7f3c480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 0x7f3c480a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3c50031440 tx=0x7f3c50005b50 comp rx=0 tx=0).stop 2024-04-23T18:27:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 msgr2=0x7f3c60198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800eaa0 tx=0x7f3c5800ef60 comp rx=0 tx=0).stop 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 shutdown_connections 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f3c480a3d20 0x7f3c480a61e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3c601985e0 0x7f3c60198a90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3c60107280 0x7f3c6019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 --2- 172.21.15.26:0/2625510327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c60102240 0x7f3c6019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 >> 172.21.15.26:0/2625510327 conn(0x7f3c600790e0 msgr2=0x7f3c60100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 shutdown_connections 2024-04-23T18:27:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.137+0000 7f3c66719640 1 -- 172.21.15.26:0/2625510327 wait complete. 2024-04-23T18:27:03.141 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --cancel --all' 2024-04-23T18:27:03.142 DEBUG:teuthology.orchestra.run.smithi026:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --cancel --all 2024-04-23T18:27:03.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 -- 172.21.15.26:0/1333388063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0101a50 msgr2=0x7f7ad0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 --2- 172.21.15.26:0/1333388063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0101a50 0x7f7ad0105a90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009900 tx=0x7f7ab802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 -- 172.21.15.26:0/1333388063 shutdown_connections 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 --2- 172.21.15.26:0/1333388063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0106170 0x7f7ad010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 --2- 172.21.15.26:0/1333388063 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0101a50 0x7f7ad0105a90 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 --2- 172.21.15.26:0/1333388063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 0x7f7ad0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 -- 172.21.15.26:0/1333388063 >> 172.21.15.26:0/1333388063 conn(0x7f7ad0077eb0 msgr2=0x7f7ad00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 -- 172.21.15.26:0/1333388063 shutdown_connections 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.256+0000 7f7ad5187640 1 -- 172.21.15.26:0/1333388063 wait complete. 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 Processor -- start 2024-04-23T18:27:03.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 -- start start 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 0x7f7ad01994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 0x7f7ad01a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad0111090 con 0x7f7ad0106170 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad0110f10 con 0x7f7ad0199f80 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ad5187640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad0111210 con 0x7f7ad01010a0 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41560/0 (socket says 172.21.15.26:41560) 2024-04-23T18:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.257+0000 7f7ac7fff640 1 -- 172.21.15.26:0/3050488355 learned_addr learned my addr 172.21.15.26:0/3050488355 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7aced9d640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 0x7f7ad01994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7acf59e640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 0x7f7ad01a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 -- 172.21.15.26:0/3050488355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 msgr2=0x7f7ad01994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 0x7f7ad01994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 -- 172.21.15.26:0/3050488355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 msgr2=0x7f7ad01a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 0x7f7ad01a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 -- 172.21.15.26:0/3050488355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad01a4d00 con 0x7f7ad0106170 2024-04-23T18:27:03.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7acf59e640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 0x7f7ad01a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8031a90 tx=0x7f7ab8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7accd99640 1 -- 172.21.15.26:0/3050488355 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab803f030 con 0x7f7ad0106170 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad01a4f90 con 0x7f7ad0106170 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7accd99640 1 -- 172.21.15.26:0/3050488355 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ab8004490 con 0x7f7ad0106170 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7accd99640 1 -- 172.21.15.26:0/3050488355 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8035070 con 0x7f7ad0106170 2024-04-23T18:27:03.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.258+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad01a53f0 con 0x7f7ad0106170 2024-04-23T18:27:03.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.260+0000 7f7accd99640 1 -- 172.21.15.26:0/3050488355 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7ab8042030 con 0x7f7ad0106170 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 Processor -- start 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 -- start start 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 0x5602c13015c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c130c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5602c1267130 con 0x5602c130c7c0 2024-04-23T18:27:03.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5602c1203c50 con 0x5602c1301b90 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5602c12630f0 con 0x5602c13011e0 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7accd99640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 0x7f7abc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41566/0 (socket says 172.21.15.26:41566) 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7ac7fff640 1 -- 172.21.15.26:0/154430567 learned_addr learned my addr 172.21.15.26:0/154430567 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7acf59e640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 0x5602c13015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7acf59e640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 0x7f7abc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.261+0000 7f7aced9d640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c130c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7accd99640 1 -- 172.21.15.26:0/3050488355 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7ab80f0e00 con 0x7f7ad0106170 2024-04-23T18:27:03.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 -- 172.21.15.26:0/154430567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 msgr2=0x5602c130c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c130c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 -- 172.21.15.26:0/154430567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 msgr2=0x5602c130cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 -- 172.21.15.26:0/154430567 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602c121dfd0 con 0x5602c13011e0 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 0x7f7abc0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8010ca0 tx=0x7f7ac8015000 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7acf59e640 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 0x5602c13015c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8013e40 tx=0x7f7ac80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97a201203b6a5bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7ac57fa640 1 -- 172.21.15.26:0/154430567 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8007b50 con 0x5602c13011e0 2024-04-23T18:27:03.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7ac57fa640 1 -- 172.21.15.26:0/154430567 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac800ac10 con 0x5602c13011e0 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.262+0000 7f7ac57fa640 1 -- 172.21.15.26:0/154430567 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8005610 con 0x5602c13011e0 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- 172.21.15.26:0/154430567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 msgr2=0x5602c13015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 0x5602c13015c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8013e40 tx=0x7f7ac80107a0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- 172.21.15.26:0/154430567 shutdown_connections 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c130c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c130c7c0 0x5602c130cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/154430567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c13011e0 0x5602c13015c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- 172.21.15.26:0/154430567 >> 172.21.15.26:0/154430567 conn(0x5602c12fd620 msgr2=0x5602c12fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- 172.21.15.26:0/154430567 shutdown_connections 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- 172.21.15.26:0/154430567 wait complete. 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 -- start start 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.263+0000 7f7ad7f79740 1 Processor -- start 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c13106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x5602c13136e0 con 0x5602c13011e0 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5602c1201640 con 0x5602c1301b90 2024-04-23T18:27:03.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x5602c1313860 con 0x5602c130c7c0 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 10 client.0 _ll_register_callbacks cb 0x5602c1201030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41580/0 (socket says 172.21.15.26:41580) 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 -- 172.21.15.26:0/1147505239 learned_addr learned my addr 172.21.15.26:0/1147505239 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c13106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7acf59e640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 -- 172.21.15.26:0/1147505239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 msgr2=0x5602c1310bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 -- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 msgr2=0x5602c13106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c13106b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 -- 172.21.15.26:0/1147505239 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c1045a20 con 0x5602c13011e0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7acf59e640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0010850 tx=0x7f7ac0010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567655d78c3736 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ab67fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000fb30 con 0x5602c13011e0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7ac000fcd0 con 0x5602c13011e0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ac800ac10 con 0x5602c13011e0 2024-04-23T18:27:03.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.264+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7ac0005020 con 0x5602c13011e0 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 10 client.16905 fetch_fsmap learned FSMap version 49 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5602c10a61b0 con 0x5602c13011e0 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac800ac10 con 0x5602c13011e0 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7ac0019410 con 0x5602c13011e0 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 10 client.16905 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 10 client.16905 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 20 client.16905 populate_metadata read hostname 'smithi026' 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c1323270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 4 client.16905 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602c1274b50 con 0x5602c1320df0 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.265+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c1323270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c1323270 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1212884735180db6 server_cookie=5220f2df19cfac4d in_seq=0 out_seq=0 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ab67fc640 10 client.16905 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1147505239 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602c1274b50 con 0x5602c1320df0 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ab67fc640 10 client.16905 handle_command_reply: tid=0 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 client.16905 shutdown 2024-04-23T18:27:03.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 msgr2=0x5602c1310170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0010850 tx=0x7f7ac0010d10 comp rx=0 tx=0).stop 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 shutdown_connections 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c1323270 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c1301b90 0x5602c13106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ab67fc640 0 client.16905 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1147505239 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c13011e0 0x5602c1310170 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.266+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 >> 172.21.15.26:0/1147505239 conn(0x5602c12fd620 msgr2=0x5602c1303490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.267+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 shutdown_connections 2024-04-23T18:27:03.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.267+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1147505239 wait complete. 2024-04-23T18:27:03.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.267+0000 7f7ad7f79740 20 client.16905 trim_cache size 0 max 16384 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 Processor -- start 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 -- start start 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c131c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01ae280 con 0x5602c1301b90 2024-04-23T18:27:03.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01aba90 con 0x5602c13011e0 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01a9900 con 0x5602c130c7c0 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41584/0 (socket says 172.21.15.26:41584) 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7ac7fff640 1 -- 172.21.15.26:0/695511903 learned_addr learned my addr 172.21.15.26:0/695511903 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.288+0000 7f7acf59e640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7aced9d640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c131c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 -- 172.21.15.26:0/695511903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 msgr2=0x5602c1310840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 -- 172.21.15.26:0/695511903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 msgr2=0x5602c131c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c131c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 -- 172.21.15.26:0/695511903 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602c1301930 con 0x5602c1301b90 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7aced9d640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c131c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80bca40 tx=0x7f7ab803b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e29606c844e1783 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac4ff9640 1 -- 172.21.15.26:0/695511903 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8049040 con 0x5602c1301b90 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac4ff9640 1 -- 172.21.15.26:0/695511903 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ab80c10f0 con 0x5602c1301b90 2024-04-23T18:27:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.289+0000 7f7ac4ff9640 1 -- 172.21.15.26:0/695511903 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab80d6110 con 0x5602c1301b90 2024-04-23T18:27:03.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- 172.21.15.26:0/695511903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 msgr2=0x5602c130dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80bca40 tx=0x7f7ab803b6b0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- 172.21.15.26:0/695511903 shutdown_connections 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c1310840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c130dbb0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/695511903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c131c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- 172.21.15.26:0/695511903 >> 172.21.15.26:0/695511903 conn(0x5602c12fd620 msgr2=0x5602c1310130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- 172.21.15.26:0/695511903 shutdown_connections 2024-04-23T18:27:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- 172.21.15.26:0/695511903 wait complete. 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 -- start start 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.290+0000 7f7ad7f79740 1 Processor -- start 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c1328720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c13863b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01a9f50 con 0x5602c1301b90 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01ab3c0 con 0x5602c13011e0 2024-04-23T18:27:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad01aa620 con 0x5602c130c7c0 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 10 client.0 _ll_register_callbacks cb 0x7f7ac0010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:41592/0 (socket says 172.21.15.26:41592) 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 -- 172.21.15.26:0/1762908497 learned_addr learned my addr 172.21.15.26:0/1762908497 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c1328720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7acf59e640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c13863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 -- 172.21.15.26:0/1762908497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 msgr2=0x5602c13863b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c13863b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 -- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 msgr2=0x5602c1328720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c1328720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 -- 172.21.15.26:0/1762908497 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c0f39310 con 0x5602c1301b90 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ac7fff640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80b91a0 tx=0x7f7ab8003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31d1a746af6ac37 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ab67fc640 10 client.0 ms_handle_connect on v2:172.21.15.26:3300/0 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab80db030 con 0x5602c1301b90 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mon.0 v2:172.21.15.26:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7ab80d6110 con 0x5602c1301b90 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ab80c10f0 con 0x5602c1301b90 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7ab8049040 con 0x5602c1301b90 2024-04-23T18:27:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 10 client.16911 fetch_fsmap learned FSMap version 49 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.291+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5602c13879a0 con 0x5602c1301b90 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mon.0 v2:172.21.15.26:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ab80c10f0 con 0x5602c1301b90 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mon.0 v2:172.21.15.26:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7ab80c55b0 con 0x5602c1301b90 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 10 client.16911 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 10 client.16911 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 20 client.16911 populate_metadata read hostname 'smithi026' 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c138a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 4 client.16911 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 --> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5602c126e9c0 con 0x5602c1320df0 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.292+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c138a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ab67fc640 10 client.16911 ms_handle_connect on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7aced9d640 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c138a130 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1b218960027666 server_cookie=fe029cdd430a8737 in_seq=0 out_seq=0 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ab67fc640 1 -- 172.21.15.26:0/1762908497 <== mds.0 v2:172.21.15.110:6838/3619972803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x5602c126e9c0 con 0x5602c1320df0 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ab67fc640 10 client.16911 handle_command_reply: tid=0 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 client.16911 shutdown 2024-04-23T18:27:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 msgr2=0x5602c1328c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80b91a0 tx=0x7f7ab8003f30 comp rx=0 tx=0).stop 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 shutdown_connections 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:6838/3619972803,v1:172.21.15.110:6839/3619972803] conn(0x5602c1320df0 0x5602c138a130 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5602c130c7c0 0x5602c13863b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x5602c1301b90 0x5602c1328c60 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ab67fc640 0 client.16911 ms_handle_reset on v2:172.21.15.110:6838/3619972803 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 --2- 172.21.15.26:0/1762908497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602c13011e0 0x5602c1328720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 >> 172.21.15.26:0/1762908497 conn(0x5602c12fd620 msgr2=0x5602c12fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 shutdown_connections 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 1 -- 172.21.15.26:0/1762908497 wait complete. 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.293+0000 7f7ad7f79740 20 client.16911 trim_cache size 0 max 16384 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 49, 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "leader": 24503, 2024-04-23T18:27:03.296 INFO:teuthology.orchestra.run.smithi026.stdout: "set_version": 44, 2024-04-23T18:27:03.297 INFO:teuthology.orchestra.run.smithi026.stdout: "sets": {} 2024-04-23T18:27:03.297 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 msgr2=0x7f7abc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 0x7f7abc0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8010ca0 tx=0x7f7ac8015000 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 msgr2=0x7f7ad01999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8031a90 tx=0x7f7ab8004270 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 shutdown_connections 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f7abc0a3d20 0x7f7abc0a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ad0199f80 0x7f7ad01a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7ad0106170 0x7f7ad01999f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 --2- 172.21.15.26:0/3050488355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad01010a0 0x7f7ad01994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 >> 172.21.15.26:0/3050488355 conn(0x7f7ad0077eb0 msgr2=0x7f7ad00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 shutdown_connections 2024-04-23T18:27:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:03.294+0000 7f7ad5187640 1 -- 172.21.15.26:0/3050488355 wait complete. 2024-04-23T18:27:03.300 INFO:tasks.quiescer:done joining Quiescers 2024-04-23T18:27:03.300 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-23T18:27:03.311 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:27:03.319 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-23T18:27:03.330 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-23T18:27:03.331 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:27:03.331 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:27:03.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-23T18:27:03.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:03 smithi026 ceph-mon[29364]: pgmap v306: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-23T18:27:03.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-23T18:27:03.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:03 smithi110 ceph-mon[32585]: pgmap v306: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-23T18:27:03.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:03 smithi131 ceph-mon[33041]: pgmap v306: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-23T18:27:04.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:04 smithi110 ceph-mon[32585]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:04.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:04 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2625510327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:04.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:04 smithi026 ceph-mon[29364]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:04.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:04 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2625510327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:04.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:04 smithi131 ceph-mon[33041]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:04.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:04 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2625510327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:05.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:05 smithi110 ceph-mon[32585]: pgmap v307: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-23T18:27:05.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:05 smithi026 ceph-mon[29364]: pgmap v307: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-23T18:27:05.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:05 smithi131 ceph-mon[33041]: pgmap v307: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-23T18:27:07.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:07 smithi110 ceph-mon[32585]: pgmap v308: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T18:27:07.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:07 smithi026 ceph-mon[29364]: pgmap v308: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T18:27:07.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:07 smithi131 ceph-mon[33041]: pgmap v308: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T18:27:09.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:09 smithi110 ceph-mon[32585]: pgmap v309: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:09 smithi026 ceph-mon[29364]: pgmap v309: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:09.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:09 smithi131 ceph-mon[33041]: pgmap v309: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:11.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:11 smithi110 ceph-mon[32585]: pgmap v310: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:11.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:11 smithi026 ceph-mon[29364]: pgmap v310: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:11.454 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:27:11.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:11 smithi131 ceph-mon[33041]: pgmap v310: 97 pgs: 97 active+clean; 360 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 951 KiB/s wr, 108 op/s 2024-04-23T18:27:11.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/540351818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac1031e0 msgr2=0x7ff9ac1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/540351818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac1035c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009900 tx=0x7ff9a802a510 comp rx=0 tx=0).stop 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/540351818 shutdown_connections 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/540351818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac10c520 0x7ff9ac10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/540351818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac103b00 0x7ff9ac10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/540351818 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac1035c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/540351818 >> 172.21.15.26:0/540351818 conn(0x7ff9ac077fd0 msgr2=0x7ff9ac0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/540351818 shutdown_connections 2024-04-23T18:27:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/540351818 wait complete. 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 Processor -- start 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.569+0000 7ff9b3aa0640 1 -- start start 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac103b00 0x7ff9ac19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac1121b0 con 0x7ff9ac10c520 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac112030 con 0x7ff9ac103b00 2024-04-23T18:27:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac112330 con 0x7ff9ac1031e0 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49534/0 (socket says 172.21.15.26:49534) 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 -- 172.21.15.26:0/2061772097 learned_addr learned my addr 172.21.15.26:0/2061772097 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b183c640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b103b640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac103b00 0x7ff9ac19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 -- 172.21.15.26:0/2061772097 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac1031e0 msgr2=0x7ff9ac19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 -- 172.21.15.26:0/2061772097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac103b00 msgr2=0x7ff9ac19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac103b00 0x7ff9ac19e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 -- 172.21.15.26:0/2061772097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ac198d40 con 0x7ff9ac10c520 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b203d640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009fd0 tx=0x7ff9a400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4017070 con 0x7ff9ac10c520 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9a4003e20 con 0x7ff9ac10c520 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4010460 con 0x7ff9ac10c520 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ac199030 con 0x7ff9ac10c520 2024-04-23T18:27:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.570+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ac1a52d0 con 0x7ff9ac10c520 2024-04-23T18:27:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.572+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9ac04fce0 con 0x7ff9ac10c520 2024-04-23T18:27:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.573+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff9a4003970 con 0x7ff9ac10c520 2024-04-23T18:27:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.573+0000 7ff9a2ffd640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 0x7ff9980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.573+0000 7ff9b183c640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 0x7ff9980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.573+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7ff9a40d0490 con 0x7ff9ac10c520 2024-04-23T18:27:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.574+0000 7ff9b183c640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 0x7ff9980a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8031440 tx=0x7ff9a8005b50 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.578+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9a409c4c0 con 0x7ff9ac10c520 2024-04-23T18:27:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.788+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ff9ac062390 con 0x7ff9980a3dc0 2024-04-23T18:27:11.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.796+0000 7ff9a2ffd640 1 -- 172.21.15.26:0/2061772097 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7ff9ac062390 con 0x7ff9980a3dc0 2024-04-23T18:27:11.789 INFO:teuthology.orchestra.run.smithi026.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:17:10", "first": "2024-04-23T18:18:00", "last": "2024-04-23T18:27:00", "last_pruned": "2024-04-23T18:27:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-23T18:27:11.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 msgr2=0x7ff9980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 0x7ff9980a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8031440 tx=0x7ff9a8005b50 comp rx=0 tx=0).stop 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 msgr2=0x7ff9ac1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009fd0 tx=0x7ff9a400b260 comp rx=0 tx=0).stop 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 shutdown_connections 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7ff9980a3dc0 0x7ff9980a6280 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9ac103b00 0x7ff9ac19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff9ac10c520 0x7ff9ac1985c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 --2- 172.21.15.26:0/2061772097 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9ac1031e0 0x7ff9ac19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 >> 172.21.15.26:0/2061772097 conn(0x7ff9ac077fd0 msgr2=0x7ff9ac0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 shutdown_connections 2024-04-23T18:27:11.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.799+0000 7ff9b3aa0640 1 -- 172.21.15.26:0/2061772097 wait complete. 2024-04-23T18:27:11.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:27:11.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 -- 172.21.15.26:0/1572369067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 msgr2=0x7fe2741025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/1572369067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741025b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe270009900 tx=0x7fe27002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:11.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 -- 172.21.15.26:0/1572369067 shutdown_connections 2024-04-23T18:27:11.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/1572369067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe274102b80 0x7fe274106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/1572369067 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2741072a0 0x7fe27410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/1572369067 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741025b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 -- 172.21.15.26:0/1572369067 >> 172.21.15.26:0/1572369067 conn(0x7fe2740793e0 msgr2=0x7fe2740797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 -- 172.21.15.26:0/1572369067 shutdown_connections 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.930+0000 7fe27b8a0640 1 -- 172.21.15.26:0/1572369067 wait complete. 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 Processor -- start 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 -- start start 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 0x7fe274198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 0x7fe27419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe274112210 con 0x7fe2741021d0 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe274112090 con 0x7fe2741072a0 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27b8a0640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe274112390 con 0x7fe27419cf60 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49550/0 (socket says 172.21.15.26:49550) 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 -- 172.21.15.26:0/4228026256 learned_addr learned my addr 172.21.15.26:0/4228026256 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:11.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe278e3b640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 0x7fe274198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe279e3d640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 0x7fe27419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 -- 172.21.15.26:0/4228026256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 msgr2=0x7fe27419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 0x7fe27419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 -- 172.21.15.26:0/4228026256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 msgr2=0x7fe274198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 0x7fe274198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe27963c640 1 -- 172.21.15.26:0/4228026256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27419da50 con 0x7fe2741021d0 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe278e3b640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 0x7fe274198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.931+0000 7fe279e3d640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 0x7fe27419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe27963c640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe270005940 tx=0x7fe270004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:11.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27003f030 con 0x7fe2741021d0 2024-04-23T18:27:11.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe270003aa0 con 0x7fe2741021d0 2024-04-23T18:27:11.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27419dce0 con 0x7fe2741021d0 2024-04-23T18:27:11.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270035070 con 0x7fe2741021d0 2024-04-23T18:27:11.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.932+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2741a5500 con 0x7fe2741021d0 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.933+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe27404fce0 con 0x7fe2741021d0 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.934+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe270042030 con 0x7fe2741021d0 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.934+0000 7fe26a7fc640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 0x7fe2600a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.935+0000 7fe278e3b640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 0x7fe2600a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.935+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7fe2700f1130 con 0x7fe2741021d0 2024-04-23T18:27:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.935+0000 7fe278e3b640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 0x7fe2600a6300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe274101800 tx=0x7fe264005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:11.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:11.938+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2700bd750 con 0x7fe2741021d0 2024-04-23T18:27:12.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.149+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 --> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe274102b80 con 0x7fe2600a3e40 2024-04-23T18:27:12.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.157+0000 7fe26a7fc640 1 -- 172.21.15.26:0/4228026256 <== mgr.15297 v2:172.21.15.26:6800/2566950754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe274102b80 con 0x7fe2600a3e40 2024-04-23T18:27:12.150 INFO:teuthology.orchestra.run.smithi026.stdout:/ 1m 6m3h 2024-04-23T18:27:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 msgr2=0x7fe2600a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 0x7fe2600a6300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe274101800 tx=0x7fe264005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 msgr2=0x7fe2741986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe270005940 tx=0x7fe270004270 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 shutdown_connections 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe2600a3e40 0x7fe2600a6300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2741072a0 0x7fe274198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.159+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe27419cf60 0x7fe27419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.160+0000 7fe27b8a0640 1 --2- 172.21.15.26:0/4228026256 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe2741021d0 0x7fe2741986f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.160+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 >> 172.21.15.26:0/4228026256 conn(0x7fe2740793e0 msgr2=0x7fe2740ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.160+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 shutdown_connections 2024-04-23T18:27:12.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.160+0000 7fe27b8a0640 1 -- 172.21.15.26:0/4228026256 wait complete. 2024-04-23T18:27:12.163 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-23T18:27:12.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:27:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 -- 172.21.15.26:0/1165372547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f58103b00 msgr2=0x7f2f5810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/1165372547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f58103b00 0x7f2f5810bfe0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009900 tx=0x7f2f4402a510 comp rx=0 tx=0).stop 2024-04-23T18:27:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 -- 172.21.15.26:0/1165372547 shutdown_connections 2024-04-23T18:27:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/1165372547 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f5810c520 0x7f2f5810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/1165372547 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f58103b00 0x7f2f5810bfe0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/1165372547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 0x7f2f581035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 -- 172.21.15.26:0/1165372547 >> 172.21.15.26:0/1165372547 conn(0x7f2f58077f50 msgr2=0x7f2f58078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 -- 172.21.15.26:0/1165372547 shutdown_connections 2024-04-23T18:27:12.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.286+0000 7f2f5eefd640 1 -- 172.21.15.26:0/1165372547 wait complete. 2024-04-23T18:27:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.287+0000 7f2f5eefd640 1 Processor -- start 2024-04-23T18:27:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.287+0000 7f2f5eefd640 1 -- start start 2024-04-23T18:27:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.287+0000 7f2f5eefd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 0x7f2f58100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.287+0000 7f2f5eefd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.287+0000 7f2f5eefd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 0x7f2f581018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5eefd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58112200 con 0x7f2f5810c520 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5eefd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58112080 con 0x7f2f581031e0 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5eefd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58112380 con 0x7f2f58103b00 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5d49a640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 0x7f2f581018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.26:49524/0 (socket says 172.21.15.26:49524) 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 -- 172.21.15.26:0/686227485 learned_addr learned my addr 172.21.15.26:0/686227485 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5cc99640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 0x7f2f58100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 -- 172.21.15.26:0/686227485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 msgr2=0x7f2f58100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 0x7f2f58100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 -- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 msgr2=0x7f2f581018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 0x7f2f581018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f57fff640 1 -- 172.21.15.26:0/686227485 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f581a6cc0 con 0x7f2f58103b00 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.288+0000 7f2f5d49a640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 0x7f2f581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:12.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.289+0000 7f2f57fff640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f44031440 tx=0x7f2f44003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.289+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4403f070 con 0x7f2f58103b00 2024-04-23T18:27:12.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.289+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f581a6f50 con 0x7f2f58103b00 2024-04-23T18:27:12.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.290+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f44031b50 con 0x7f2f58103b00 2024-04-23T18:27:12.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.290+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581a73c0 con 0x7f2f58103b00 2024-04-23T18:27:12.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.290+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44035030 con 0x7f2f58103b00 2024-04-23T18:27:12.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.291+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3c005290 con 0x7f2f58103b00 2024-04-23T18:27:12.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.294+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2f44042070 con 0x7f2f58103b00 2024-04-23T18:27:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.295+0000 7f2f55ffb640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 0x7f2f480a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.295+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7f2f440f1390 con 0x7f2f58103b00 2024-04-23T18:27:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.295+0000 7f2f5cc99640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 0x7f2f480a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.296+0000 7f2f5cc99640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 0x7f2f480a62a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009b60 tx=0x7f2f4c005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.299+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f440bda10 con 0x7f2f58103b00 2024-04-23T18:27:12.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:12 smithi110 ceph-mon[32585]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:12.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:12 smithi026 ceph-mon[29364]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:12.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.493+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f3c005850 con 0x7f2f58103b00 2024-04-23T18:27:12.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.495+0000 7f2f55ffb640 1 -- 172.21.15.26:0/686227485 <== mon.1 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+23194 (secure 0 0 0) 0x7f2f440bd310 con 0x7f2f58103b00 2024-04-23T18:27:12.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:12.488 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":97,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:27:11.102468+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:27:12.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 msgr2=0x7f2f480a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 0x7f2f480a62a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009b60 tx=0x7f2f4c005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:12.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 msgr2=0x7f2f58101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f44031440 tx=0x7f2f44003b90 comp rx=0 tx=0).stop 2024-04-23T18:27:12.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 shutdown_connections 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2f480a3de0 0x7f2f480a62a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2f5810c520 0x7f2f581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f58103b00 0x7f2f58101370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 --2- 172.21.15.26:0/686227485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f581031e0 0x7f2f58100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 >> 172.21.15.26:0/686227485 conn(0x7f2f58077f50 msgr2=0x7f2f580ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 shutdown_connections 2024-04-23T18:27:12.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.497+0000 7f2f5eefd640 1 -- 172.21.15.26:0/686227485 wait complete. 2024-04-23T18:27:12.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:12 smithi131 ceph-mon[33041]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:12.501 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:27:12.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:27:12.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 -- 172.21.15.26:0/701272037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 msgr2=0x7f54c810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/701272037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c810ee40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009ca0 tx=0x7f54c002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 -- 172.21.15.26:0/701272037 shutdown_connections 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/701272037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806b1f0 0x7f54c8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/701272037 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c810ee40 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/701272037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c806a840 0x7f54c806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 -- 172.21.15.26:0/701272037 >> 172.21.15.26:0/701272037 conn(0x7f54c8077eb0 msgr2=0x7f54c80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 -- 172.21.15.26:0/701272037 shutdown_connections 2024-04-23T18:27:12.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.614+0000 7f54cd1cd640 1 -- 172.21.15.26:0/701272037 wait complete. 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 Processor -- start 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 -- start start 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806a840 0x7f54c8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 0x7f54c819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81121d0 con 0x7f54c8107270 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8112050 con 0x7f54c806a840 2024-04-23T18:27:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54cd1cd640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8112350 con 0x7f54c819cff0 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54bffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54bffff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49598/0 (socket says 172.21.15.26:49598) 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54bffff640 1 -- 172.21.15.26:0/4222929757 learned_addr learned my addr 172.21.15.26:0/4222929757 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54c6d9d640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806a840 0x7f54c8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.615+0000 7f54c759e640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 0x7f54c819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 -- 172.21.15.26:0/4222929757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 msgr2=0x7f54c819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 0x7f54c819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 -- 172.21.15.26:0/4222929757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806a840 msgr2=0x7f54c8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806a840 0x7f54c8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 -- 172.21.15.26:0/4222929757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c819dae0 con 0x7f54c8107270 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54c759e640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 0x7f54c819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54bffff640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f54b000d770 tx=0x7f54b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0017070 con 0x7f54c8107270 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b0009c30 con 0x7f54c8107270 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c819ddd0 con 0x7f54c8107270 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b000f620 con 0x7f54c8107270 2024-04-23T18:27:12.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.616+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81a5610 con 0x7f54c8107270 2024-04-23T18:27:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.617+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c80691c0 con 0x7f54c8107270 2024-04-23T18:27:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.618+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f54b001a070 con 0x7f54c8107270 2024-04-23T18:27:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.619+0000 7f54c4d99640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 0x7f54b40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.619+0000 7f54c6d9d640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 0x7f54b40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.619+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7f54b00d0330 con 0x7f54c8107270 2024-04-23T18:27:12.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.619+0000 7f54c6d9d640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 0x7f54b40a6260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fd0 tx=0x7f54b8005850 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.622+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54b0009de0 con 0x7f54c8107270 2024-04-23T18:27:12.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.809+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54c810a560 con 0x7f54c8107270 2024-04-23T18:27:12.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.809+0000 7f54c4d99640 1 -- 172.21.15.26:0/4222929757 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f54b009c9d0 con 0x7f54c8107270 2024-04-23T18:27:12.802 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:27:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 msgr2=0x7f54b40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 0x7f54b40a6260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fd0 tx=0x7f54b8005850 comp rx=0 tx=0).stop 2024-04-23T18:27:12.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 msgr2=0x7f54c8198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f54b000d770 tx=0x7f54b000dc30 comp rx=0 tx=0).stop 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 shutdown_connections 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f54b40a3da0 0x7f54b40a6260 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c819cff0 0x7f54c819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f54c8107270 0x7f54c8198c70 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 --2- 172.21.15.26:0/4222929757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c806a840 0x7f54c8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 >> 172.21.15.26:0/4222929757 conn(0x7f54c8077eb0 msgr2=0x7f54c80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.811+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 shutdown_connections 2024-04-23T18:27:12.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.812+0000 7f54cd1cd640 1 -- 172.21.15.26:0/4222929757 wait complete. 2024-04-23T18:27:12.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:27:12.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 -- 172.21.15.26:0/3159116556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48105dd0 msgr2=0x7f0d481061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/3159116556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48105dd0 0x7f0d481061b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009900 tx=0x7f0d4002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 -- 172.21.15.26:0/3159116556 shutdown_connections 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/3159116556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48101a40 0x7f0d4810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/3159116556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d481010a0 0x7f0d48101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/3159116556 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48105dd0 0x7f0d481061b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 -- 172.21.15.26:0/3159116556 >> 172.21.15.26:0/3159116556 conn(0x7f0d48077c40 msgr2=0x7f0d48078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 -- 172.21.15.26:0/3159116556 shutdown_connections 2024-04-23T18:27:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 -- 172.21.15.26:0/3159116556 wait complete. 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.928+0000 7f0d4fee1640 1 Processor -- start 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 -- start start 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 0x7f0d48198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d48105dd0 0x7f0d4819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0d48111150 con 0x7f0d48101a40 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d48110fd0 con 0x7f0d481010a0 2024-04-23T18:27:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4fee1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481112d0 con 0x7f0d48105dd0 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49608/0 (socket says 172.21.15.26:49608) 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 -- 172.21.15.26:0/1747974166 learned_addr learned my addr 172.21.15.26:0/1747974166 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4dc7d640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 0x7f0d48198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 -- 172.21.15.26:0/1747974166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d48105dd0 msgr2=0x7f0d4819d010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d48105dd0 0x7f0d4819d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 -- 172.21.15.26:0/1747974166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 msgr2=0x7f0d48198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 0x7f0d48198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 -- 172.21.15.26:0/1747974166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4819d740 con 0x7f0d48101a40 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4dc7d640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 0x7f0d48198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d4d47c640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800d950 tx=0x7f0d3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.929+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38017070 con 0x7f0d48101a40 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.930+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d38009c30 con 0x7f0d48101a40 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.930+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4819da30 con 0x7f0d48101a40 2024-04-23T18:27:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.930+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3800f620 con 0x7f0d48101a40 2024-04-23T18:27:12.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.930+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d481a52c0 con 0x7f0d48101a40 2024-04-23T18:27:12.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.930+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d48102540 con 0x7f0d48101a40 2024-04-23T18:27:12.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.932+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0d38005030 con 0x7f0d48101a40 2024-04-23T18:27:12.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.933+0000 7f0d3effd640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 0x7f0d340a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:12.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.933+0000 7f0d4dc7d640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 0x7f0d340a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:12.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.933+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7f0d380d0470 con 0x7f0d48101a40 2024-04-23T18:27:12.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.934+0000 7f0d4dc7d640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 0x7f0d340a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d40031440 tx=0x7f0d40005a20 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:12.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:12.937+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d3809cb70 con 0x7f0d48101a40 2024-04-23T18:27:13.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.123+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0d481a5580 con 0x7f0d48101a40 2024-04-23T18:27:13.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.123+0000 7f0d3effd640 1 -- 172.21.15.26:0/1747974166 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7f0d38009de0 con 0x7f0d48101a40 2024-04-23T18:27:13.116 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:27:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 msgr2=0x7f0d340a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 0x7f0d340a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d40031440 tx=0x7f0d40005a20 comp rx=0 tx=0).stop 2024-04-23T18:27:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 msgr2=0x7f0d48198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800d950 tx=0x7f0d3800de10 comp rx=0 tx=0).stop 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 shutdown_connections 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d340a3da0 0x7f0d340a6260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d48105dd0 0x7f0d4819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d48101a40 0x7f0d48198c90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 --2- 172.21.15.26:0/1747974166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d481010a0 0x7f0d48198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 >> 172.21.15.26:0/1747974166 conn(0x7f0d48077c40 msgr2=0x7f0d48104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:13.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 shutdown_connections 2024-04-23T18:27:13.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.125+0000 7f0d4fee1640 1 -- 172.21.15.26:0/1747974166 wait complete. 2024-04-23T18:27:13.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:27:13.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 -- 172.21.15.26:0/547125845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 msgr2=0x7f25d4102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 --2- 172.21.15.26:0/547125845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d4102570 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009900 tx=0x7f25c402a410 comp rx=0 tx=0).stop 2024-04-23T18:27:13.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 -- 172.21.15.26:0/547125845 shutdown_connections 2024-04-23T18:27:13.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 --2- 172.21.15.26:0/547125845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4107260 0x7f25d410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 --2- 172.21.15.26:0/547125845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4102b40 0x7f25d4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 --2- 172.21.15.26:0/547125845 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d4102570 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 -- 172.21.15.26:0/547125845 >> 172.21.15.26:0/547125845 conn(0x7f25d40fd990 msgr2=0x7f25d40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 -- 172.21.15.26:0/547125845 shutdown_connections 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.241+0000 7f25d9115640 1 -- 172.21.15.26:0/547125845 wait complete. 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 Processor -- start 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 -- start start 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4107260 0x7f25d419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4198610 0x7f25d4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41121c0 con 0x7f25d4102190 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4112040 con 0x7f25d4198610 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d9115640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4112340 con 0x7f25d4107260 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d2d9d640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49626/0 (socket says 172.21.15.26:49626) 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d2d9d640 1 -- 172.21.15.26:0/4039941091 learned_addr learned my addr 172.21.15.26:0/4039941091 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:13.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.242+0000 7f25d359e640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4198610 0x7f25d4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25cbfff640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4107260 0x7f25d419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 -- 172.21.15.26:0/4039941091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4107260 msgr2=0x7f25d419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4107260 0x7f25d419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 -- 172.21.15.26:0/4039941091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4198610 msgr2=0x7f25d4198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4198610 0x7f25d4198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 -- 172.21.15.26:0/4039941091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d4199130 con 0x7f25d4102190 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d2d9d640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f25c4005940 tx=0x7f25c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c403f030 con 0x7f25d4102190 2024-04-23T18:27:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c4003aa0 con 0x7f25d4102190 2024-04-23T18:27:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d41993c0 con 0x7f25d4102190 2024-04-23T18:27:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4035070 con 0x7f25d4102190 2024-04-23T18:27:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.243+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41a5560 con 0x7f25d4102190 2024-04-23T18:27:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.244+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d40691c0 con 0x7f25d4102190 2024-04-23T18:27:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.245+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f25c4042030 con 0x7f25d4102190 2024-04-23T18:27:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.246+0000 7f25d0d99640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 0x7f25c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.246+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7f25c40f1140 con 0x7f25d4102190 2024-04-23T18:27:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.246+0000 7f25cbfff640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 0x7f25c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.247+0000 7f25cbfff640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 0x7f25c00a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25d40fdeb0 tx=0x7f25bc005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.249+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25c40bd110 con 0x7f25d4102190 2024-04-23T18:27:13.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.438+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f25d4102b40 con 0x7f25d4102190 2024-04-23T18:27:13.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.438+0000 7f25d0d99640 1 -- 172.21.15.26:0/4039941091 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f25c40c1410 con 0x7f25d4102190 2024-04-23T18:27:13.431 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:27:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 msgr2=0x7f25c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 0x7f25c00a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25d40fdeb0 tx=0x7f25bc005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 msgr2=0x7f25d419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f25c4005940 tx=0x7f25c4004270 comp rx=0 tx=0).stop 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 shutdown_connections 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f25c00a3d20 0x7f25c00a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d4198610 0x7f25d4198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d4107260 0x7f25d419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 --2- 172.21.15.26:0/4039941091 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f25d4102190 0x7f25d419e410 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 >> 172.21.15.26:0/4039941091 conn(0x7f25d40fd990 msgr2=0x7f25d40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:13.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 shutdown_connections 2024-04-23T18:27:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.440+0000 7f25d9115640 1 -- 172.21.15.26:0/4039941091 wait complete. 2024-04-23T18:27:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:13 smithi110 ceph-mon[32585]: pgmap v312: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:13 smithi110 ceph-mon[32585]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:13 smithi110 ceph-mon[32585]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/686227485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:13.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:13 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4222929757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:13.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:13 smithi026 ceph-mon[29364]: pgmap v312: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:13.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:13 smithi026 ceph-mon[29364]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:13 smithi026 ceph-mon[29364]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/686227485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:13.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:13 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4222929757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:13.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-23T18:27:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:13 smithi131 ceph-mon[33041]: pgmap v312: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:13 smithi131 ceph-mon[33041]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:13 smithi131 ceph-mon[33041]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:27:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/686227485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:13.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:13 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4222929757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 -- 172.21.15.26:0/205990168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 msgr2=0x7f607810c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 --2- 172.21.15.26:0/205990168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607810c030 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6068009900 tx=0x7f606802a480 comp rx=0 tx=0).stop 2024-04-23T18:27:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 -- 172.21.15.26:0/205990168 shutdown_connections 2024-04-23T18:27:13.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 --2- 172.21.15.26:0/205990168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 0x7f607810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 --2- 172.21.15.26:0/205990168 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607810c030 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 --2- 172.21.15.26:0/205990168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 0x7f6078103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 -- 172.21.15.26:0/205990168 >> 172.21.15.26:0/205990168 conn(0x7f6078077fb0 msgr2=0x7f60780783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 -- 172.21.15.26:0/205990168 shutdown_connections 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.557+0000 7f607fd29640 1 -- 172.21.15.26:0/205990168 wait complete. 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 Processor -- start 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 -- start start 2024-04-23T18:27:13.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 0x7f607819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 0x7f6078198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f6078112280 con 0x7f6078103b50 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6078112100 con 0x7f607810c570 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607fd29640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f6078112400 con 0x7f6078103230 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:49646/0 (socket says 172.21.15.26:49646) 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 -- 172.21.15.26:0/3135758044 learned_addr learned my addr 172.21.15.26:0/3135758044 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:13.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 -- 172.21.15.26:0/3135758044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 msgr2=0x7f607819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607dac5640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 0x7f607819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 0x7f607819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 -- 172.21.15.26:0/3135758044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 msgr2=0x7f6078198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607e2c6640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 0x7f6078198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 0x7f6078198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.558+0000 7f607d2c4640 1 -- 172.21.15.26:0/3135758044 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6078198da0 con 0x7f6078103b50 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f607d2c4640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6068005ab0 tx=0x7f6068003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606803f070 con 0x7f6078103b50 2024-04-23T18:27:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6068031d60 con 0x7f6078103b50 2024-04-23T18:27:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6078199030 con 0x7f6078103b50 2024-04-23T18:27:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068035030 con 0x7f6078103b50 2024-04-23T18:27:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.559+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60781a52e0 con 0x7f6078103b50 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.560+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60780691c0 con 0x7f6078103b50 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.561+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6068042070 con 0x7f6078103b50 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.562+0000 7f606effd640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 0x7f60640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.562+0000 7f607dac5640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 0x7f60640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.562+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8557+0+0 (secure 0 0 0) 0x7f60680f1160 con 0x7f6078103b50 2024-04-23T18:27:13.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.562+0000 7f607dac5640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 0x7f60640a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f607400a810 tx=0x7f6074008040 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.566+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60680bd780 con 0x7f6078103b50 2024-04-23T18:27:13.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.808+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f60781076f0 con 0x7f6078103b50 2024-04-23T18:27:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.817+0000 7f606effd640 1 -- 172.21.15.26:0/3135758044 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746060 (secure 0 0 0) 0x7f60680bd080 con 0x7f6078103b50 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 26, 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "active_gid": 15297, 2024-04-23T18:27:13.820 INFO:teuthology.orchestra.run.smithi026.stdout: "active_name": "x", 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "active_addrs": { 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6800", 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2566950754 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6801", 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2566950754 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "active_addr": "172.21.15.26:6801/2566950754", 2024-04-23T18:27:13.822 INFO:teuthology.orchestra.run.smithi026.stdout: "active_change": "2024-04-23T18:17:09.436088+0000", 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "active_mgr_features": 4540701547738038271, 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "available": true, 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "standbys": [ 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "gid": 15291, 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "y", 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:27:13.823 INFO:teuthology.orchestra.run.smithi026.stdout: "available_modules": [ 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "alerts", 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.824 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.825 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.826 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.827 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.828 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.829 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.830 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.831 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.832 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.833 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_destination": { 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_destination", 2024-04-23T18:27:13.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:27:13.835 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_from_name": { 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_from_name", 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "Ceph", 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email From: name", 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_host": { 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_host", 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.838 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP server", 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_password": { 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_password", 2024-04-23T18:27:13.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Password to authenticate with", 2024-04-23T18:27:13.840 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_port": { 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_port", 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "465", 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP port", 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_sender": { 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_sender", 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_ssl": { 2024-04-23T18:27:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_ssl", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_user": { 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_user", 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "User to authenticate as", 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "balancer", 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "active": { 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "active", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_time": { 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_time", 2024-04-23T18:27:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0000", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_weekday": { 2024-04-23T18:27:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_weekday", 2024-04-23T18:27:13.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:27:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_max_iterations": { 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "25", 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "250", 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_metrics": { 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_metrics", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_step": { 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_step", 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.5", 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0.001", 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "0.999", 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "end_time": { 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_time", 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2359", 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "end_weekday": { 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_weekday", 2024-04-23T18:27:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.866 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.868 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.869 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.870 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "min_score": { 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_score", 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:13.871 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.872 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "upmap", 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.873 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-compat", 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "none", 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "read", 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap", 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap-read" 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.874 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Balancer mode", 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_ids": { 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_ids", 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.875 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.876 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.877 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_deviation": { 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_deviation", 2024-04-23T18:27:13.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:27:13.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_optimizations": { 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.880 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm", 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_down_multiplier": { 2024-04-23T18:27:13.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_down_multiplier", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:27:13.883 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_refresh_rate": { 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_refresh_rate", 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "20", 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_starting_port": { 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_starting_port", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4721", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.886 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_ptrace": { 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_ptrace", 2024-04-23T18:27:13.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.888 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_interval": { 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_interval", 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.889 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:27:13.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.7", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm_log_destination": { 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm_log_destination", 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.892 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "file", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "file,syslog", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "syslog" 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.893 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "cgroups_split": { 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cgroups_split", 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.894 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.895 INFO:teuthology.orchestra.run.smithi026.stdout: "config_checks_enabled": { 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_checks_enabled", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.896 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "config_dashboard": { 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_dashboard", 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.897 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.898 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_alertmanager": { 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_alertmanager", 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.899 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_base": { 2024-04-23T18:27:13.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_base", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:27:13.901 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_elasticsearch": { 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.902 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "elasticsearch container image", 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_grafana": { 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_grafana", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.904 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_haproxy": { 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_haproxy", 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.905 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "HAproxy container image", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.906 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_agent": { 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.907 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.908 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_collector": { 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.909 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_query": { 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.910 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger query container image", 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.911 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_keepalived": { 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_keepalived", 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.912 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Keepalived container image", 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_loki": { 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_loki", 2024-04-23T18:27:13.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Loki container image", 2024-04-23T18:27:13.914 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_node_exporter": { 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_node_exporter", 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_nvmeof": { 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_nvmeof", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.917 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Nvme-of container image", 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_prometheus": { 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_prometheus", 2024-04-23T18:27:13.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.919 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_promtail": { 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_promtail", 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:27:13.920 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Promtail container image", 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_samba": { 2024-04-23T18:27:13.921 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_samba", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:27:13.922 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_snmp_gateway": { 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.923 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.924 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "container_init": { 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_init", 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.925 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "daemon_cache_timeout": { 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.927 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:27:13.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "900", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "default_registry": { 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_registry", 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.930 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io", 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "device_cache_timeout": { 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_cache_timeout", 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1800", 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.932 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "device_enhanced_scan": { 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_enhanced_scan", 2024-04-23T18:27:13.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.934 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "facts_cache_timeout": { 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "facts_cache_timeout", 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:13.935 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.936 INFO:teuthology.orchestra.run.smithi026.stdout: "grafana_dashboards_path": { 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.937 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "host_check_interval": { 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "host_check_interval", 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.938 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.939 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "hw_monitoring": { 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hw_monitoring", 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.940 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "inventory_list_all": { 2024-04-23T18:27:13.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "inventory_list_all", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.942 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.943 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.944 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "log_refresh_metadata": { 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_refresh_metadata", 2024-04-23T18:27:13.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:27:13.946 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.947 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.949 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.950 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.951 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.952 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.953 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.954 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "*", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.955 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "max_count_per_host": { 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_count_per_host", 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.956 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.957 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd_draining_count": { 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_osd_draining_count", 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:13.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.959 INFO:teuthology.orchestra.run.smithi026.stdout: "migration_current": { 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "migration_current", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.960 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "internal - do not modify", 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.961 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "root", 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm-package", 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: "root" 2024-04-23T18:27:13.962 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "oob_default_addr": { 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "oob_default_addr", 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.963 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "169.254.1.1", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.964 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "prometheus_alerts_path": { 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.965 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_insecure": { 2024-04-23T18:27:13.966 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_insecure", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.967 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_password": { 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_password", 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.969 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_url": { 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_url", 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.970 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.971 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_username": { 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_username", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.972 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "secure_monitoring_stack": { 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.973 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.974 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "service_discovery_port": { 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "service_discovery_port", 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8765", 2024-04-23T18:27:13.975 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "ssh_config_file": { 2024-04-23T18:27:13.976 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssh_config_file", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:27:13.977 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "use_agent": { 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_agent", 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.978 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.979 INFO:teuthology.orchestra.run.smithi026.stdout: "use_repo_digest": { 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_repo_digest", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.980 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_failed_host_check": { 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.981 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.982 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_daemons": { 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_hosts": { 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:27:13.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crash", 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:13.986 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.987 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.988 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.991 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:13.992 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.993 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.994 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "retain_interval": { 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retain_interval", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "31536000", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.995 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_recent_interval": { 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_recent_interval", 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:13.996 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1209600", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:13.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dashboard", 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:13.998 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:13.999 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:27:14.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.001 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.002 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.004 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:27:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:27:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.019 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:27:14.020 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.021 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.022 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.023 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.025 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_URL": { 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.027 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.030 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.031 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.032 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.033 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.034 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.035 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.036 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "[]", 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.038 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:27:14.039 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "{}", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.040 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.042 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.043 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:27:14.044 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.045 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.048 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.049 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.050 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:27:14.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.053 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.055 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.058 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.060 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:27:14.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8", 2024-04-23T18:27:14.063 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.064 INFO:teuthology.orchestra.run.smithi026.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "45", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.065 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.068 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:27:14.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.070 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.072 INFO:teuthology.orchestra.run.smithi026.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.074 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.075 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.076 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:27:14.077 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.078 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cross_origin_url": { 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cross_origin_url", 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.079 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.080 INFO:teuthology.orchestra.run.smithi026.stdout: "crt_file": { 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crt_file", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.081 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "debug": { 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "debug", 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "jwt_token_ttl": { 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "jwt_token_ttl", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "28800", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.084 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.085 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.086 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.088 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.090 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.091 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.092 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.093 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.094 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "motd": { 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "motd", 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "The message of the day", 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.096 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.097 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:14.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.099 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8080", 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.102 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl_server_port": { 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl_server_port", 2024-04-23T18:27:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8443", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.105 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "redirect", 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect" 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.106 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.107 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: "url_prefix": { 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url_prefix", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.109 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.110 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "devicehealth", 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_monitoring": { 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_monitoring", 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.111 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "monitor device health metrics", 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.112 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.113 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.117 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.118 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.119 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.120 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "mark_out_threshold": { 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mark_out_threshold", 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2419200", 2024-04-23T18:27:14.121 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": { 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_name", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "device_health_metrics", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.123 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "retention_period": { 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retention_period", 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.124 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15552000", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.125 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_frequency": { 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_frequency", 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.126 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "self_heal": { 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "self_heal", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.128 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.130 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_threshold": { 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_threshold", 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7257600", 2024-04-23T18:27:14.131 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "diskprediction_local", 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.133 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.134 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.136 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.138 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.140 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "predict_interval": { 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predict_interval", 2024-04-23T18:27:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.142 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "predictor_model": { 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predictor_model", 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.143 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "prophetstor", 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.145 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "influx", 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": false, 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "batch_size": { 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "batch_size", 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5000", 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "database": { 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "database", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "ceph", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.149 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "hostname": { 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hostname", 2024-04-23T18:27:14.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:27:14.151 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.152 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "30", 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "5", 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.154 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.155 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.156 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.157 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.158 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.159 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.160 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "password": { 2024-04-23T18:27:14.161 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "password", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:27:14.162 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "port": { 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "port", 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.163 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8086", 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server port", 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.164 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "false", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.165 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "threads": { 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threads", 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.166 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "32", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.167 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "username": { 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "username", 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.168 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "verify_ssl": { 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "verify_ssl", 2024-04-23T18:27:14.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "true", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:14.170 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "insights", 2024-04-23T18:27:14.171 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.172 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.173 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.175 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.176 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.177 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.178 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.179 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "iostat", 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.180 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.181 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.182 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.185 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.186 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.188 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "k8sevents", 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.189 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_event_retention_days": { 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.190 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "config_check_secs": { 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_check_secs", 2024-04-23T18:27:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "10", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:27:14.192 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.193 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.194 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.195 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.196 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.198 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.199 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.200 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "localpool", 2024-04-23T18:27:14.201 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "failure_domain": { 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "failure_domain", 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "host", 2024-04-23T18:27:14.202 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.203 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.204 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.205 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.206 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.208 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.210 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": { 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_size", 2024-04-23T18:27:14.211 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:27:14.212 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "num_rep": { 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "num_rep", 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3", 2024-04-23T18:27:14.213 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.214 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": { 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_num", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "128", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.215 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "prefix": { 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prefix", 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.216 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "subtree": { 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "subtree", 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "rack", 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.218 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mds_autoscaler", 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.220 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.221 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.222 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.223 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.224 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.225 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.226 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.227 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.228 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirroring", 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.229 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.230 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.231 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.232 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.233 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.234 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.235 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.236 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "nfs", 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.237 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.238 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.239 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.242 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.243 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "fail_fs": { 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "fail_fs", 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator": { 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm", 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "rook", 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "test_orchestrator" 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Orchestrator backend", 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_perf_query", 2024-04-23T18:27:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_support", 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_autoscaler", 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "threshold": { 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threshold", 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1.0", 2024-04-23T18:27:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "scaling threshold", 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "progress", 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_pg_recovery_event": { 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "max_completed_events": { 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_completed_events", 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "50", 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus", 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cache", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "exclude_perf_counters": { 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "exclude_perf_counters", 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools": { 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "300", 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_interval": { 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_interval", 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15.0", 2024-04-23T18:27:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:27:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "9283", 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "stale_cache_strategy": { 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stale_cache_strategy", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "log", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "default", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "default", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "error" 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_support", 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_snap_create": { 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "trash_purge_schedule": { 2024-04-23T18:27:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "trash_purge_schedule", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "restful", 2024-04-23T18:27:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_auth": { 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_auth", 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rgw", 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rook", 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "storage_class": { 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "storage_class", 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "local", 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "selftest", 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.379 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.380 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.381 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.383 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "roption1": { 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption1", 2024-04-23T18:27:14.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.385 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "roption2": { 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption2", 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.386 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "xyz", 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.387 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption1": { 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption1", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.388 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption2": { 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption2", 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.389 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.390 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption3": { 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption3", 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.391 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption4": { 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption4", 2024-04-23T18:27:14.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.393 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption5": { 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption5", 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.394 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.395 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption6": { 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption6", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.396 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption7": { 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption7", 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.397 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "42", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.398 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "testkey": { 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testkey", 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.399 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.400 INFO:teuthology.orchestra.run.smithi026.stdout: "testlkey": { 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testlkey", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.401 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "testnewline": { 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testnewline", 2024-04-23T18:27:14.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snap_schedule", 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.404 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_m_granularity": { 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_m_granularity", 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.405 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.406 INFO:teuthology.orchestra.run.smithi026.stdout: "dump_on_update": { 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dump_on_update", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.407 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.408 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.409 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.411 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.413 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.414 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.415 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stats", 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.417 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.418 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.420 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.422 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.423 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.424 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "status", 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.427 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.429 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.430 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.431 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.432 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.433 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.434 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telegraf", 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "address": { 2024-04-23T18:27:14.435 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "address", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.436 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.437 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.438 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.439 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.440 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.441 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.442 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.443 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.444 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.445 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.446 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telemetry", 2024-04-23T18:27:14.447 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_basic": { 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_basic", 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.448 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.449 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_crash": { 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_crash", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.450 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_device": { 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_device", 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.451 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.452 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_ident": { 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_ident", 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.453 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.454 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_perf": { 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_perf", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.455 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "contact": { 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "contact", 2024-04-23T18:27:14.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.457 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "description": { 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "description", 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.458 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "device_url": { 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_url", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.460 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.461 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.462 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "24", 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "8", 2024-04-23T18:27:14.463 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.465 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.465 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.465 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.465 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "last_opt_revision": { 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "last_opt_revision", 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1", 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.466 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard": { 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard", 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.467 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard_description": { 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard_description", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.469 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.471 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.473 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.474 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.475 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.476 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "organization": { 2024-04-23T18:27:14.477 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "organization", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.478 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "proxy": { 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "proxy", 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.479 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.480 INFO:teuthology.orchestra.run.smithi026.stdout: "url": { 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.481 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.482 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "test_orchestrator", 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.483 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.484 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.486 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.487 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.488 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.489 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.490 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "volumes", 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.491 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.492 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.493 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.494 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.495 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.496 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.497 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.499 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_clones": { 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_clones", 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.500 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4", 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.501 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.502 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.502 INFO:teuthology.orchestra.run.smithi026.stdout: "periodic_async_work": { 2024-04-23T18:27:14.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "periodic_async_work", 2024-04-23T18:27:14.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.503 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Periodically check for async work", 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_delay": { 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:27:14.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.506 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix", 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "discovery_interval": { 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "discovery_interval", 2024-04-23T18:27:14.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "100", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.509 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "identifier": { 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "identifier", 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.510 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.511 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.512 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.516 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.518 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.519 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.520 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_host": { 2024-04-23T18:27:14.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_host", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.522 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_port": { 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_port", 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.523 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10051", 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.524 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_sender": { 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_sender", 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.525 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: "gid": 24886, 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "z", 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: "available_modules": [ 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.527 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "alerts", 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.528 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.530 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.531 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.532 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.534 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.535 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.536 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.537 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_destination": { 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_destination", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.538 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_from_name": { 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_from_name", 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.539 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "Ceph", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email From: name", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.540 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_host": { 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_host", 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.541 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP server", 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.542 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_password": { 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_password", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.543 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Password to authenticate with", 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_port": { 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_port", 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.544 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "465", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP port", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_sender": { 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_sender", 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.546 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_ssl": { 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_ssl", 2024-04-23T18:27:14.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.548 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_user": { 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_user", 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.549 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "User to authenticate as", 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "balancer", 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "active": { 2024-04-23T18:27:14.551 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "active", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:27:14.552 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_time": { 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_time", 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.553 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0000", 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.554 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_weekday": { 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_weekday", 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:14.555 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_max_iterations": { 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:27:14.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "25", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "250", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.557 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_metrics": { 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_metrics", 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:27:14.558 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_step": { 2024-04-23T18:27:14.559 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_step", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.5", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0.001", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "0.999", 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.560 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "end_time": { 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_time", 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.561 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2359", 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.562 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "end_weekday": { 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_weekday", 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:14.563 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.565 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.566 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.567 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.569 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.570 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.571 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "min_score": { 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_score", 2024-04-23T18:27:14.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.573 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.574 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "upmap", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-compat", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "none", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "read", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap", 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap-read" 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.575 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Balancer mode", 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_ids": { 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_ids", 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.576 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.578 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_deviation": { 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_deviation", 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.579 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.580 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_optimizations": { 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.581 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.582 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm", 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_down_multiplier": { 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_down_multiplier", 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.583 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.584 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_refresh_rate": { 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_refresh_rate", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "20", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.585 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_starting_port": { 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_starting_port", 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.586 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4721", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.587 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_ptrace": { 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_ptrace", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.588 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_interval": { 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_interval", 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.589 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:27:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.591 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.7", 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.592 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm_log_destination": { 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm_log_destination", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "file", 2024-04-23T18:27:14.593 INFO:teuthology.orchestra.run.smithi026.stdout: "file,syslog", 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "syslog" 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.594 INFO:teuthology.orchestra.run.smithi026.stdout: "cgroups_split": { 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cgroups_split", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.595 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "config_checks_enabled": { 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_checks_enabled", 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.597 INFO:teuthology.orchestra.run.smithi026.stdout: "config_dashboard": { 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_dashboard", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.598 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_alertmanager": { 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_alertmanager", 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_base": { 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_base", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.601 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_elasticsearch": { 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:27:14.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "elasticsearch container image", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.603 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_grafana": { 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_grafana", 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.604 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_haproxy": { 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_haproxy", 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.606 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "HAproxy container image", 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_agent": { 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:27:14.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:27:14.608 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_collector": { 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.609 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.610 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_query": { 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.611 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger query container image", 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_keepalived": { 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_keepalived", 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.612 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Keepalived container image", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.613 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_loki": { 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_loki", 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:27:14.614 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Loki container image", 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_node_exporter": { 2024-04-23T18:27:14.615 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_node_exporter", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:14.616 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_nvmeof": { 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_nvmeof", 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.617 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Nvme-of container image", 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_prometheus": { 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_prometheus", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.619 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_promtail": { 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_promtail", 2024-04-23T18:27:14.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Promtail container image", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.621 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_samba": { 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_samba", 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.622 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_snmp_gateway": { 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.624 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "container_init": { 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_init", 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.625 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "daemon_cache_timeout": { 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.627 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.628 INFO:teuthology.orchestra.run.smithi026.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "900", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.629 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "default_registry": { 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_registry", 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.630 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.631 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "device_cache_timeout": { 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_cache_timeout", 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.632 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1800", 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.633 INFO:teuthology.orchestra.run.smithi026.stdout: "device_enhanced_scan": { 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_enhanced_scan", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.634 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "facts_cache_timeout": { 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "facts_cache_timeout", 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.635 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "grafana_dashboards_path": { 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.637 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "host_check_interval": { 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "host_check_interval", 2024-04-23T18:27:14.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.639 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "hw_monitoring": { 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hw_monitoring", 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.640 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "inventory_list_all": { 2024-04-23T18:27:14.641 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "inventory_list_all", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:27:14.642 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.643 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.644 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "log_refresh_metadata": { 2024-04-23T18:27:14.645 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_refresh_metadata", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.647 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.648 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.649 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.650 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.651 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.652 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.653 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "*", 2024-04-23T18:27:14.654 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "max_count_per_host": { 2024-04-23T18:27:14.655 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_count_per_host", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:27:14.656 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd_draining_count": { 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_osd_draining_count", 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.657 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.658 INFO:teuthology.orchestra.run.smithi026.stdout: "migration_current": { 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "migration_current", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.659 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "internal - do not modify", 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.660 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "root", 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm-package", 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "root" 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.661 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "oob_default_addr": { 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "oob_default_addr", 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.662 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "169.254.1.1", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.663 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "prometheus_alerts_path": { 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.664 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_insecure": { 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_insecure", 2024-04-23T18:27:14.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.666 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_password": { 2024-04-23T18:27:14.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_password", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.671 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_url": { 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_url", 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_username": { 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_username", 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.674 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "secure_monitoring_stack": { 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:27:14.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:27:14.676 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "service_discovery_port": { 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "service_discovery_port", 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.677 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8765", 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "ssh_config_file": { 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssh_config_file", 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.679 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "use_agent": { 2024-04-23T18:27:14.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_agent", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:27:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "use_repo_digest": { 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_repo_digest", 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_failed_host_check": { 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.684 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_daemons": { 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.685 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.686 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_hosts": { 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.687 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.688 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crash", 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.689 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.690 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.691 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.692 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.693 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.694 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.695 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.696 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "retain_interval": { 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retain_interval", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "31536000", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.697 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_recent_interval": { 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_recent_interval", 2024-04-23T18:27:14.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1209600", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.699 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dashboard", 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.700 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.701 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:27:14.702 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.705 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.706 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:27:14.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.709 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:27:14.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.711 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.712 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:27:14.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.714 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.715 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:27:14.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.717 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.718 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.719 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.720 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.721 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.722 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.723 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:27:14.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_URL": { 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.726 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.727 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.728 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.731 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:27:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.733 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.734 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.735 INFO:teuthology.orchestra.run.smithi026.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "[]", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.736 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.737 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "{}", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.739 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.740 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.741 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.742 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:27:14.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.744 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.745 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:27:14.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.747 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.748 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.749 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.750 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.751 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:27:14.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.753 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.754 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.755 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:27:14.757 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.759 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "45", 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.761 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.763 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.764 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:27:14.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.766 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.767 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.768 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.769 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.770 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.771 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.772 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "cross_origin_url": { 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cross_origin_url", 2024-04-23T18:27:14.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.774 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "crt_file": { 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crt_file", 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.775 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.776 INFO:teuthology.orchestra.run.smithi026.stdout: "debug": { 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "debug", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.777 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "jwt_token_ttl": { 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "jwt_token_ttl", 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.778 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "28800", 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.780 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.781 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.782 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.784 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.785 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.786 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.788 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "motd": { 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "motd", 2024-04-23T18:27:14.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "The message of the day", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.790 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.791 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.792 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.793 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.794 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8080", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.796 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl_server_port": { 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl_server_port", 2024-04-23T18:27:14.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8443", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.798 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "redirect", 2024-04-23T18:27:14.799 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect" 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.800 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:14.801 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "url_prefix": { 2024-04-23T18:27:14.802 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url_prefix", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.803 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "devicehealth", 2024-04-23T18:27:14.804 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_monitoring": { 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_monitoring", 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.805 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "monitor device health metrics", 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.806 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.807 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.808 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.809 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.811 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.812 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.813 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.814 INFO:teuthology.orchestra.run.smithi026.stdout: "mark_out_threshold": { 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mark_out_threshold", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2419200", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:27:14.815 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": { 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_name", 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.816 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "device_health_metrics", 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "retention_period": { 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retention_period", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15552000", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.818 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_frequency": { 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_frequency", 2024-04-23T18:27:14.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.820 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "self_heal": { 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "self_heal", 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.821 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_threshold": { 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_threshold", 2024-04-23T18:27:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7257600", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:27:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "diskprediction_local", 2024-04-23T18:27:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "predict_interval": { 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predict_interval", 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "predictor_model": { 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predictor_model", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "prophetstor", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "influx", 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": false, 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "batch_size": { 2024-04-23T18:27:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "batch_size", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5000", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "database": { 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "database", 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "ceph", 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "hostname": { 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hostname", 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "30", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "5", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:27:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "password": { 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "password", 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.855 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "port": { 2024-04-23T18:27:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "port", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8086", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server port", 2024-04-23T18:27:14.857 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.858 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "false", 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "threads": { 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threads", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "32", 2024-04-23T18:27:14.860 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "username": { 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "username", 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.861 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "verify_ssl": { 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "verify_ssl", 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "true", 2024-04-23T18:27:14.863 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "insights", 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.865 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.866 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.867 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.868 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.870 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.871 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.873 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.873 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.873 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.873 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.875 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "iostat", 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.877 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.878 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.880 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.881 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.882 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.883 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.884 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "k8sevents", 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.885 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_event_retention_days": { 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.886 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "config_check_secs": { 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_check_secs", 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.887 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "10", 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.889 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.890 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.891 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.893 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.894 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.895 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "localpool", 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "failure_domain": { 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "failure_domain", 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.897 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "host", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": { 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_size", 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "num_rep": { 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "num_rep", 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3", 2024-04-23T18:27:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": { 2024-04-23T18:27:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_num", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "128", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "prefix": { 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prefix", 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "subtree": { 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "subtree", 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "rack", 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.914 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mds_autoscaler", 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.916 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.917 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.918 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.919 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.920 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.921 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.922 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.923 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.924 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirroring", 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.925 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.926 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.928 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.930 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.931 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.932 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "nfs", 2024-04-23T18:27:14.933 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.934 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.935 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.936 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.937 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.938 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.939 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.941 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.942 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "fail_fs": { 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "fail_fs", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.944 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.945 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.946 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.947 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.948 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.949 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.950 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.951 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator": { 2024-04-23T18:27:14.952 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm", 2024-04-23T18:27:14.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rook", 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: "test_orchestrator" 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Orchestrator backend", 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_perf_query", 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.956 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.957 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.959 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.960 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.961 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.962 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_support", 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.964 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.965 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.966 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.967 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.968 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.969 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.970 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.971 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_autoscaler", 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.972 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.973 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.974 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.976 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.978 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.979 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.980 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.981 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.982 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "threshold": { 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threshold", 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1.0", 2024-04-23T18:27:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "scaling threshold", 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "progress", 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_pg_recovery_event": { 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.995 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "max_completed_events": { 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_completed_events", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "50", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.996 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:14.997 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:14.998 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus", 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:14.999 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cache", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.000 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "exclude_perf_counters": { 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "exclude_perf_counters", 2024-04-23T18:27:15.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:27:15.002 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools": { 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools", 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "300", 2024-04-23T18:27:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_interval": { 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_interval", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15.0", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "9283", 2024-04-23T18:27:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "stale_cache_strategy": { 2024-04-23T18:27:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stale_cache_strategy", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "log", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "default", 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "default", 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "error" 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_support", 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_snap_create": { 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.035 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "trash_purge_schedule": { 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "trash_purge_schedule", 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.036 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.037 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "restful", 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.038 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_auth": { 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_auth", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.039 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.040 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.041 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.042 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.043 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.045 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.046 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.047 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.049 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.050 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.052 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rgw", 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.055 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.057 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.058 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.059 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.061 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rook", 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.064 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.065 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.066 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.067 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.068 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.070 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "storage_class": { 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "storage_class", 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.071 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "local", 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "selftest", 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.074 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.075 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.077 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.078 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.080 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.082 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "roption1": { 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption1", 2024-04-23T18:27:15.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.084 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "roption2": { 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption2", 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "xyz", 2024-04-23T18:27:15.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.086 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption1": { 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption1", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.087 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption2": { 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption2", 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.088 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption3": { 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption3", 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.090 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption4": { 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption4", 2024-04-23T18:27:15.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.092 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption5": { 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption5", 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.093 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.094 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption6": { 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption6", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.095 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption7": { 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption7", 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.096 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "42", 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "testkey": { 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testkey", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.098 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "testlkey": { 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testlkey", 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "testnewline": { 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testnewline", 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snap_schedule", 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_m_granularity": { 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_m_granularity", 2024-04-23T18:27:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "dump_on_update": { 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dump_on_update", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stats", 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "status", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telegraf", 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "address": { 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "address", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telemetry", 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_basic": { 2024-04-23T18:27:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_basic", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_crash": { 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_crash", 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_device": { 2024-04-23T18:27:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_device", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:27:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_ident": { 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_ident", 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_perf": { 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_perf", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "contact": { 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "contact", 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "description": { 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "description", 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "device_url": { 2024-04-23T18:27:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_url", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "24", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "8", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_opt_revision": { 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "last_opt_revision", 2024-04-23T18:27:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard": { 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard", 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard_description": { 2024-04-23T18:27:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard_description", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "organization": { 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "organization", 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "proxy": { 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "proxy", 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "url": { 2024-04-23T18:27:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "test_orchestrator", 2024-04-23T18:27:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "volumes", 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_clones": { 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_clones", 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4", 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "periodic_async_work": { 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "periodic_async_work", 2024-04-23T18:27:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Periodically check for async work", 2024-04-23T18:27:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_delay": { 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix", 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "discovery_interval": { 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "discovery_interval", 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "100", 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "identifier": { 2024-04-23T18:27:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "identifier", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.205 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.206 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.207 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.208 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.209 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.210 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.211 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.212 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.213 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.214 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.215 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_host": { 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_host", 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.216 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_port": { 2024-04-23T18:27:15.217 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_port", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10051", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.218 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_sender": { 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_sender", 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.220 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "modules": [ 2024-04-23T18:27:15.221 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm", 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "dashboard", 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "iostat", 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nfs", 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "restful", 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_schedule" 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "available_modules": [ 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.222 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "alerts", 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.223 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.225 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.227 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.228 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.229 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.230 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.231 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.232 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_destination": { 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_destination", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.233 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_from_name": { 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_from_name", 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.234 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "Ceph", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Email From: name", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.235 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_host": { 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_host", 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.236 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP server", 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_password": { 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_password", 2024-04-23T18:27:15.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Password to authenticate with", 2024-04-23T18:27:15.238 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_port": { 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_port", 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "465", 2024-04-23T18:27:15.239 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP port", 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.240 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_sender": { 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_sender", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.241 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_ssl": { 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_ssl", 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.242 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "smtp_user": { 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "smtp_user", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.244 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "User to authenticate as", 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "balancer", 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "active": { 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "active", 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.246 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.247 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_time": { 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_time", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0000", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.248 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "begin_weekday": { 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "begin_weekday", 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.249 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.250 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_max_iterations": { 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "25", 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:15.251 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "250", 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_metrics": { 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_metrics", 2024-04-23T18:27:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:27:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_compat_step": { 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crush_compat_step", 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.254 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.5", 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0.001", 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "0.999", 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "end_time": { 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_time", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2359", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.256 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "end_weekday": { 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "end_weekday", 2024-04-23T18:27:15.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "0", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "6", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:27:15.258 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.259 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.260 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.261 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.262 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.264 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.266 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "min_score": { 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_score", 2024-04-23T18:27:15.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.269 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "upmap", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-compat", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "none", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "read", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap", 2024-04-23T18:27:15.270 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap-read" 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Balancer mode", 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_ids": { 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_ids", 2024-04-23T18:27:15.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.273 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_deviation": { 2024-04-23T18:27:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_deviation", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:27:15.275 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "upmap_max_optimizations": { 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.276 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm", 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.278 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_down_multiplier": { 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_down_multiplier", 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.279 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.281 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.281 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:27:15.281 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_refresh_rate": { 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_refresh_rate", 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "20", 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "agent_starting_port": { 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "agent_starting_port", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4721", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.284 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_ptrace": { 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_ptrace", 2024-04-23T18:27:15.285 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:27:15.286 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_interval": { 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_interval", 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:15.287 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:27:15.288 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0.7", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:27:15.289 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm_log_destination": { 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cephadm_log_destination", 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.290 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "file", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "file,syslog", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "syslog" 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.291 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "cgroups_split": { 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cgroups_split", 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.292 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "config_checks_enabled": { 2024-04-23T18:27:15.293 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_checks_enabled", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:27:15.294 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "config_dashboard": { 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_dashboard", 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.295 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_alertmanager": { 2024-04-23T18:27:15.296 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_alertmanager", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:15.297 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_base": { 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_base", 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:27:15.298 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_elasticsearch": { 2024-04-23T18:27:15.299 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "elasticsearch container image", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.300 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_grafana": { 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_grafana", 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.301 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_haproxy": { 2024-04-23T18:27:15.302 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_haproxy", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "HAproxy container image", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_agent": { 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.304 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_collector": { 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:27:15.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_jaeger_query": { 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.307 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Jaeger query container image", 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_keepalived": { 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_keepalived", 2024-04-23T18:27:15.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Keepalived container image", 2024-04-23T18:27:15.309 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_loki": { 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_loki", 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.310 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Loki container image", 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.311 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_node_exporter": { 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_node_exporter", 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.312 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_nvmeof": { 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_nvmeof", 2024-04-23T18:27:15.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Nvme-of container image", 2024-04-23T18:27:15.314 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_prometheus": { 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_prometheus", 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.315 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Prometheus container image", 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_promtail": { 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_promtail", 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.317 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Promtail container image", 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_samba": { 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_samba", 2024-04-23T18:27:15.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.319 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image_snmp_gateway": { 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.320 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "container_init": { 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "container_init", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.322 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "daemon_cache_timeout": { 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.323 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.324 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "900", 2024-04-23T18:27:15.325 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "default_registry": { 2024-04-23T18:27:15.326 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "default_registry", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "docker.io", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:27:15.327 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "device_cache_timeout": { 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_cache_timeout", 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1800", 2024-04-23T18:27:15.328 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "device_enhanced_scan": { 2024-04-23T18:27:15.329 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_enhanced_scan", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:27:15.330 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "facts_cache_timeout": { 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "facts_cache_timeout", 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.331 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "grafana_dashboards_path": { 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.333 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "host_check_interval": { 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "host_check_interval", 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "hw_monitoring": { 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hw_monitoring", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.336 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "inventory_list_all": { 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "inventory_list_all", 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.337 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.338 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.339 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.340 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "log_refresh_metadata": { 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_refresh_metadata", 2024-04-23T18:27:15.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:27:15.342 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.343 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:27:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "*", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "max_count_per_host": { 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_count_per_host", 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd_draining_count": { 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_osd_draining_count", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "migration_current": { 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "migration_current", 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "internal - do not modify", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "mode": { 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mode", 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "root", 2024-04-23T18:27:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm-package", 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "root" 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "oob_default_addr": { 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "oob_default_addr", 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "169.254.1.1", 2024-04-23T18:27:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "prometheus_alerts_path": { 2024-04-23T18:27:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_insecure": { 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_insecure", 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_password": { 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_password", 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_url": { 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_url", 2024-04-23T18:27:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "registry_username": { 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "registry_username", 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "secure_monitoring_stack": { 2024-04-23T18:27:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:27:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "service_discovery_port": { 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "service_discovery_port", 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8765", 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "ssh_config_file": { 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssh_config_file", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "use_agent": { 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_agent", 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "use_repo_digest": { 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "use_repo_digest", 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_failed_host_check": { 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.376 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.377 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_daemons": { 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.378 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_on_stray_hosts": { 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:27:15.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.380 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crash", 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.381 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.382 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.383 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.384 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.385 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.386 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.387 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.388 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "retain_interval": { 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retain_interval", 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "31536000", 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.390 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_recent_interval": { 2024-04-23T18:27:15.391 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_recent_interval", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1209600", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:27:15.392 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dashboard", 2024-04-23T18:27:15.393 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.394 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.395 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.396 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.398 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.399 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.400 INFO:teuthology.orchestra.run.smithi026.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.401 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.402 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.403 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.404 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:27:15.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.406 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.407 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:27:15.408 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.409 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.410 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.411 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.412 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.413 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.415 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:27:15.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.417 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.418 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_URL": { 2024-04-23T18:27:15.419 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.421 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.422 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.423 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.424 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.425 INFO:teuthology.orchestra.run.smithi026.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.426 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.427 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:27:15.428 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "[]", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.429 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.430 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "{}", 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.431 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.432 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.433 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.435 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.436 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.438 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.439 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.440 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.441 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:27:15.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.443 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.444 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:27:15.445 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.446 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.447 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:27:15.448 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.449 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.450 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8", 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.451 INFO:teuthology.orchestra.run.smithi026.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "45", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.452 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.453 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "admin", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.455 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.456 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.458 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.459 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.460 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.461 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "cross_origin_url": { 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cross_origin_url", 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.466 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "crt_file": { 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "crt_file", 2024-04-23T18:27:15.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.468 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "debug": { 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "debug", 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.469 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "jwt_token_ttl": { 2024-04-23T18:27:15.470 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "jwt_token_ttl", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "28800", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.471 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.472 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.474 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.475 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.476 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.477 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.478 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.479 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.480 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "motd": { 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "motd", 2024-04-23T18:27:15.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "The message of the day", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.482 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:27:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:27:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.485 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.486 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:15.487 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:15.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8080", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.489 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.490 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.491 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl_server_port": { 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl_server_port", 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8443", 2024-04-23T18:27:15.492 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.493 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "redirect", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.494 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "redirect" 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.495 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:15.496 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "url_prefix": { 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url_prefix", 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.497 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.498 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "devicehealth", 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.499 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_monitoring": { 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_monitoring", 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.500 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "monitor device health metrics", 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.501 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.502 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.503 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.504 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.505 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.506 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "mark_out_threshold": { 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mark_out_threshold", 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "2419200", 2024-04-23T18:27:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": { 2024-04-23T18:27:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pool_name", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "device_health_metrics", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:27:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "retention_period": { 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "retention_period", 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15552000", 2024-04-23T18:27:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:27:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_frequency": { 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_frequency", 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "self_heal": { 2024-04-23T18:27:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "self_heal", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:27:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "warn_threshold": { 2024-04-23T18:27:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "warn_threshold", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7257600", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "diskprediction_local", 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "predict_interval": { 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predict_interval", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "86400", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.529 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "predictor_model": { 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "predictor_model", 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.530 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "prophetstor", 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "600", 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.532 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "influx", 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": false, 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "batch_size": { 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "batch_size", 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.534 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5000", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.535 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "database": { 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "database", 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "ceph", 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.536 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "hostname": { 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "hostname", 2024-04-23T18:27:15.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:27:15.538 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.539 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "30", 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "5", 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.541 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.542 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.543 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.545 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.546 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.547 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.548 INFO:teuthology.orchestra.run.smithi026.stdout: "password": { 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "password", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.549 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "port": { 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "port", 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "8086", 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "InfluxDB server port", 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.551 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "ssl": { 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ssl", 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "false", 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.552 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "threads": { 2024-04-23T18:27:15.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threads", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "32", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:27:15.554 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "username": { 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "username", 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.555 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.556 INFO:teuthology.orchestra.run.smithi026.stdout: "verify_ssl": { 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "verify_ssl", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "true", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.557 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "insights", 2024-04-23T18:27:15.558 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.559 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.560 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.563 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.564 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.566 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "iostat", 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.567 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.568 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.569 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.570 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.571 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.572 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.573 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.575 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "k8sevents", 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.576 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_event_retention_days": { 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "7", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.577 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "config_check_secs": { 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "config_check_secs", 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.578 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "10", 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.580 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.581 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.582 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.583 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.584 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.585 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.586 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "localpool", 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "failure_domain": { 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "failure_domain", 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.588 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "host", 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.589 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.590 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.591 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.592 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.594 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.595 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.596 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": { 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "min_size", 2024-04-23T18:27:15.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:27:15.598 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "num_rep": { 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "num_rep", 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3", 2024-04-23T18:27:15.599 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": { 2024-04-23T18:27:15.600 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_num", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "128", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:27:15.601 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "prefix": { 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prefix", 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "subtree": { 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "subtree", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "rack", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.604 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.605 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mds_autoscaler", 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.607 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.609 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.610 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.611 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.613 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirroring", 2024-04-23T18:27:15.614 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.617 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.618 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.619 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.621 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.622 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "nfs", 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.623 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.624 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.626 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.628 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.629 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.630 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.631 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.632 INFO:teuthology.orchestra.run.smithi026.stdout: "fail_fs": { 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "fail_fs", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.633 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.635 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.636 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.637 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.638 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.639 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.640 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.641 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator": { 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "orchestrator", 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm", 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "rook", 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "test_orchestrator" 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Orchestrator backend", 2024-04-23T18:27:15.643 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_perf_query", 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.644 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.645 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.646 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.650 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "osd_support", 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.653 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.654 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.655 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.656 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.658 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "pg_autoscaler", 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "threshold": { 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "threshold", 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "3.0", 2024-04-23T18:27:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1.0", 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "scaling threshold", 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "progress", 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_pg_recovery_event": { 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:27:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "max_completed_events": { 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_completed_events", 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "50", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "sleep_interval": { 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "sleep_interval", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "5", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "prometheus", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "cache": { 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "cache", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "exclude_perf_counters": { 2024-04-23T18:27:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "exclude_perf_counters", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:27:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.696 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.697 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.698 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools": { 2024-04-23T18:27:15.699 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.700 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.701 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "300", 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.702 INFO:teuthology.orchestra.run.smithi026.stdout: "scrape_interval": { 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "scrape_interval", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15.0", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.703 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.704 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "::", 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.705 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "9283", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.706 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "stale_cache_strategy": { 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stale_cache_strategy", 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.707 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "log", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.708 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_behaviour": { 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_behaviour", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "default", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.709 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "default", 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "error" 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.710 INFO:teuthology.orchestra.run.smithi026.stdout: "standby_error_status_code": { 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "standby_error_status_code", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "500", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "400", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "599", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.711 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_support", 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.712 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.713 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.714 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.715 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.716 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.717 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.719 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_snap_create": { 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10", 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.721 INFO:teuthology.orchestra.run.smithi026.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.722 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "trash_purge_schedule": { 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "trash_purge_schedule", 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.723 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "restful", 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_auth": { 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enable_auth", 2024-04-23T18:27:15.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.726 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "key_file": { 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "key_file", 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.727 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.729 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.730 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.731 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.732 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.733 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.735 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "server_addr": { 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_addr", 2024-04-23T18:27:15.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.737 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "server_port": { 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "server_port", 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.738 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rgw", 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.741 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.742 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.743 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.744 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.745 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.746 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.747 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rook", 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.749 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.750 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.752 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.753 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.754 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.756 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "storage_class": { 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "storage_class", 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "local", 2024-04-23T18:27:15.757 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "selftest", 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.759 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.760 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.761 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.762 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.763 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.764 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.765 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.766 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "roption1": { 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption1", 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.767 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.768 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "roption2": { 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "roption2", 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "xyz", 2024-04-23T18:27:15.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption1": { 2024-04-23T18:27:15.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption1", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.771 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption2": { 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption2", 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.772 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption3": { 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption3", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "float", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.774 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption4": { 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption4", 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.775 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption5": { 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption5", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.777 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption6": { 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption6", 2024-04-23T18:27:15.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.779 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "rwoption7": { 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rwoption7", 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.780 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "1", 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "42", 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "testkey": { 2024-04-23T18:27:15.781 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testkey", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.782 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "testlkey": { 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testlkey", 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.783 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "testnewline": { 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "testnewline", 2024-04-23T18:27:15.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.785 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snap_schedule", 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.786 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_m_granularity": { 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "allow_m_granularity", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.787 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "dump_on_update": { 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "dump_on_update", 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.788 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.789 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.790 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.791 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.792 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.794 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.795 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.796 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "stats", 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.797 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.798 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.799 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.800 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.801 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.802 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.803 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "status", 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.806 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.807 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.809 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.811 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.812 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.813 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telegraf", 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.814 INFO:teuthology.orchestra.run.smithi026.stdout: "address": { 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "address", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.815 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.816 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "15", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.817 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.818 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.819 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.820 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.822 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.823 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.824 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "telemetry", 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_basic": { 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_basic", 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.826 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.827 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_crash": { 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_crash", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.828 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_device": { 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_device", 2024-04-23T18:27:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_ident": { 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_ident", 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "channel_perf": { 2024-04-23T18:27:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "channel_perf", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:27:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "contact": { 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "contact", 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "description": { 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "description", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "device_url": { 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "device_url", 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "enabled": { 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "enabled", 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "24", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "8", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "last_opt_revision": { 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "last_opt_revision", 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "1", 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard": { 2024-04-23T18:27:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "leaderboard_description": { 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "leaderboard_description", 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "organization": { 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "organization", 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "proxy": { 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "proxy", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "url": { 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "url", 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "test_orchestrator", 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.860 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.861 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.862 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.864 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.865 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.866 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "volumes", 2024-04-23T18:27:15.867 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.868 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.871 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.872 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.873 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.874 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.875 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "max_concurrent_clones": { 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "max_concurrent_clones", 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "4", 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.876 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "periodic_async_work": { 2024-04-23T18:27:15.877 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "periodic_async_work", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.878 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Periodically check for async work", 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_delay": { 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "0", 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.880 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "True", 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.881 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix", 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "can_run": true, 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "error_string": "", 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "module_options": { 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "discovery_interval": { 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "discovery_interval", 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "uint", 2024-04-23T18:27:15.883 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "100", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "identifier": { 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "identifier", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.885 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "interval": { 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "interval", 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "secs", 2024-04-23T18:27:15.886 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "60", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.887 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "log_level": { 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_level", 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.888 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.889 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster": { 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster", 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.890 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_cluster_level": { 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_cluster_level", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "info", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [ 2024-04-23T18:27:15.892 INFO:teuthology.orchestra.run.smithi026.stdout: "", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "critical", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "debug", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "error", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "info", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "warning" 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": { 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "log_to_file", 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "bool", 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "False", 2024-04-23T18:27:15.894 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_host": { 2024-04-23T18:27:15.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_host", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.896 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_port": { 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_port", 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "int", 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.897 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "10051", 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "zabbix_sender": { 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "zabbix_sender", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "str", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "level": "advanced", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 0, 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "min": "", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "max": "", 2024-04-23T18:27:15.899 INFO:teuthology.orchestra.run.smithi026.stdout: "enum_allowed": [], 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: "desc": "", 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: "long_desc": "", 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: "tags": [], 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: "see_also": [] 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.900 INFO:teuthology.orchestra.run.smithi026.stdout: "services": { 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "dashboard": "https://172.21.15.26:8443/" 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "always_on_modules": { 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus": [ 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "balancer", 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "devicehealth", 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator", 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscaler", 2024-04-23T18:27:15.901 INFO:teuthology.orchestra.run.smithi026.stdout: "progress", 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_support", 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "status", 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "telemetry", 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "volumes" 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific": [ 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "balancer", 2024-04-23T18:27:15.902 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "devicehealth", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscaler", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "progress", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_support", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "status", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "telemetry", 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "volumes" 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy": [ 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "balancer", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "devicehealth", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscaler", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "progress", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_support", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "status", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "telemetry", 2024-04-23T18:27:15.904 INFO:teuthology.orchestra.run.smithi026.stdout: "volumes" 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "reef": [ 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "balancer", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "devicehealth", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscaler", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "progress", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_support", 2024-04-23T18:27:15.905 INFO:teuthology.orchestra.run.smithi026.stdout: "status", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "telemetry", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "volumes" 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "squid": [ 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "balancer", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "crash", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "devicehealth", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "orchestrator", 2024-04-23T18:27:15.906 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscaler", 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "progress", 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_support", 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "status", 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "telemetry", 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "volumes" 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "last_failure_osd_epoch": 89, 2024-04-23T18:27:15.907 INFO:teuthology.orchestra.run.smithi026.stdout: "active_clients": [ 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "devicehealth", 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 758224487 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "libcephsqlite", 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2685484464 2024-04-23T18:27:15.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "rbd_support", 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3705733836 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snap_schedule", 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 829648974 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "snap_schedule", 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 126766956 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "volumes", 2024-04-23T18:27:15.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:0", 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2352317355 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-04-23T18:27:15.914 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:15.916 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1747974166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.916 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4039941091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.916 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:14 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3135758044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:27:15.917 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:15 smithi131 ceph-mon[33041]: pgmap v313: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:15.917 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1747974166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.917 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4039941091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.917 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:14 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3135758044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:27:15.917 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:15 smithi110 ceph-mon[32585]: pgmap v313: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 msgr2=0x7f60640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 0x7f60640a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f607400a810 tx=0x7f6074008040 comp rx=0 tx=0).stop 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 msgr2=0x7f607819e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6068005ab0 tx=0x7f6068003cd0 comp rx=0 tx=0).stop 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 shutdown_connections 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f60640a3da0 0x7f60640a6260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f607810c570 0x7f6078198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f6078103b50 0x7f607819e960 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 --2- 172.21.15.26:0/3135758044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6078103230 0x7f607819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 >> 172.21.15.26:0/3135758044 conn(0x7f6078077fb0 msgr2=0x7f60780fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.822+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 shutdown_connections 2024-04-23T18:27:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:13.823+0000 7f607fd29640 1 -- 172.21.15.26:0/3135758044 wait complete. 2024-04-23T18:27:15.925 DEBUG:tasks.check_counter:Getting stats from a 2024-04-23T18:27:15.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-23T18:27:15.926 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1747974166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.926 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4039941091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:15.926 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:14 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3135758044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:27:15.926 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:15 smithi026 ceph-mon[29364]: pgmap v313: 97 pgs: 97 active+clean; 361 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 32 op/s 2024-04-23T18:27:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:27:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:17 smithi110 ceph-mon[32585]: pgmap v314: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 550 KiB/s wr, 33 op/s 2024-04-23T18:27:17.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:17 smithi110 ceph-mon[32585]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:17 smithi026 ceph-mon[29364]: pgmap v314: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 550 KiB/s wr, 33 op/s 2024-04-23T18:27:17.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:17 smithi026 ceph-mon[29364]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:17 smithi131 ceph-mon[33041]: pgmap v314: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 550 KiB/s wr, 33 op/s 2024-04-23T18:27:17.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:17 smithi131 ceph-mon[33041]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:18.157 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 211, 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 200, 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 1586704, 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 122784, 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.042973697, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.011475813, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.036427404, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.003912865, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 199, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.009550317, 2024-04-23T18:27:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000047991 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 1586704, 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 122784 2024-04-23T18:27:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 2, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 152, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 320, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 182832, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.012579278, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.007769688, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.002623638, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 149, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.011077133, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000074343 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 182832 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 4, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 4, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 448, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.000675892, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.000040759, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.000604441, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 3, 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000762552, 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000254184 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "cct": { 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "total_workers": 0, 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "unhealthy_workers": 0 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool": { 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_items": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_bytes": 149017, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_items": 31, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bytes": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_items": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_items": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_items": 0, 2024-04-23T18:27:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_items": 2, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_items": 0, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_items": 0 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "val": 0, 2024-04-23T18:27:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "max": 104857600, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "get_started": 0, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "get": 213, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "get_sum": 1556973, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_success": 213, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "take": 0, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "take_sum": 0, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "put": 213, 2024-04-23T18:27:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "put_sum": 1556973, 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: "wait": { 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:18.175 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:18.811 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-23T18:27:18.811 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-23T18:27:18.812 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-23T18:27:18.812 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-23T18:27:18.812 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-23T18:27:18.812 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:27:18.812 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:27:18.812 DEBUG:tasks.check_counter:Getting stats from d 2024-04-23T18:27:18.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-23T18:27:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:27:19.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:19 smithi110 ceph-mon[32585]: pgmap v316: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-04-23T18:27:19.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:19 smithi026 ceph-mon[29364]: pgmap v316: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-04-23T18:27:19.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:19 smithi131 ceph-mon[33041]: pgmap v316: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-04-23T18:27:20.843 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 0, 2024-04-23T18:27:20.843 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 2, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 0, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.001352883, 2024-04-23T18:27:20.844 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.000260674, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.845 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:20.846 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 2, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 157, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 320, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 183712, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.012747089, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.008658002, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.001838314, 2024-04-23T18:27:20.847 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 154, 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.009696965, 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000062967 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:20.848 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 183712 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 220, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 207, 2024-04-23T18:27:20.849 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 1591056, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 126432, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.043088003, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.012655780, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.036347984, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.004065322, 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:20.850 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 207, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.009375865, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000045294 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.851 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 1591056, 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 126432 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "cct": { 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "total_workers": 0, 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "unhealthy_workers": 0 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool": { 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_items": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:20.857 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_bytes": 148972, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_items": 31, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bytes": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_items": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:20.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_items": 0, 2024-04-23T18:27:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:20.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_items": 0, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_items": 2, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_items": 0, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_items": 0 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:20.860 INFO:teuthology.orchestra.run.smithi026.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "val": 0, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "max": 104857600, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "get_started": 0, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "get": 218, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "get_sum": 1557623, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_success": 218, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "take": 0, 2024-04-23T18:27:20.861 INFO:teuthology.orchestra.run.smithi026.stdout: "take_sum": 0, 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "put": 218, 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "put_sum": 1557623, 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "wait": { 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:20.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:20.863 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:21.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:21 smithi110 ceph-mon[32585]: pgmap v317: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 587 KiB/s wr, 29 op/s 2024-04-23T18:27:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:21 smithi026 ceph-mon[29364]: pgmap v317: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 587 KiB/s wr, 29 op/s 2024-04-23T18:27:21.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:21 smithi131 ceph-mon[33041]: pgmap v317: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 587 KiB/s wr, 29 op/s 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:27:21.495 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:27:21.495 DEBUG:tasks.check_counter:Getting stats from g 2024-04-23T18:27:21.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-23T18:27:21.750 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:27:23.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:23 smithi026 ceph-mon[29364]: pgmap v318: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:23.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:23 smithi110 ceph-mon[32585]: pgmap v318: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:23.457 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:23.457 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:23.457 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 213, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 202, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 1587248, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 124064, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.043213002, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.011793440, 2024-04-23T18:27:23.458 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.036051481, 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.003897014, 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 201, 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.008355925, 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000041571 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.459 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 1587248, 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 124064 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:23.460 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 2, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 154, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 320, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 183184, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.013280423, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.008870797, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.002024491, 2024-04-23T18:27:23.461 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 151, 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.008165521, 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000054076 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:23.462 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 183184 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 4, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 4, 2024-04-23T18:27:23.463 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 448, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.000765608, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.000042190, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.000714025, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:23.464 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 3, 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000813244, 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000271081 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:23.465 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "cct": { 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "total_workers": 0, 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "unhealthy_workers": 0 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.466 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool": { 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:23.467 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:23.468 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:23.469 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_items": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:23.470 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:23.471 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:23.471 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:23.471 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_bytes": 149017, 2024-04-23T18:27:23.471 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_items": 31, 2024-04-23T18:27:23.473 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bytes": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_items": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:23.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_items": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_items": 0, 2024-04-23T18:27:23.475 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_items": 2, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_items": 0, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_items": 0 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "val": 0, 2024-04-23T18:27:23.476 INFO:teuthology.orchestra.run.smithi026.stdout: "max": 104857600, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "get_started": 0, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "get": 215, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "get_sum": 1557233, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_success": 215, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "take": 0, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "take_sum": 0, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "put": 215, 2024-04-23T18:27:23.477 INFO:teuthology.orchestra.run.smithi026.stdout: "put_sum": 1557233, 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: "wait": { 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:23.478 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:23.490 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:23 smithi131 ceph-mon[33041]: pgmap v318: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:24.114 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:27:24.115 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:27:24.115 DEBUG:tasks.check_counter:Getting stats from j 2024-04-23T18:27:24.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-23T18:27:24.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:24 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:24.241 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:24 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:24.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:24 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:27:25.309 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:25 smithi026 ceph-mon[29364]: pgmap v319: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:25.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:25 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:25 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:25 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:25 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:25 smithi110 ceph-mon[32585]: pgmap v319: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:25.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:25 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:25 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:25 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:25 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:25 smithi131 ceph-mon[33041]: pgmap v319: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 520 KiB/s wr, 31 op/s 2024-04-23T18:27:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:25 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:25 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:25 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:25.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:25 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:26.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:26 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.443 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:26 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.492 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.493 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:26 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:26.630 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 0, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 2, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 0, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:26.631 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.001429232, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.000376903, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:26.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:26.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 218, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 207, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 1588608, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 127264, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:26.634 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.046192666, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.012869493, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.039924467, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.003944916, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 206, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.008862621, 2024-04-23T18:27:26.635 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000043022 2024-04-23T18:27:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:26.636 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 1588608, 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 127264 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:26.637 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_messages": 6, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages": 161, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_bytes": 3312, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_bytes": 184512, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:26.638 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_total_time": 0.013356366, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_send_time": 0.008539910, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_recv_time": 0.002552449, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 159, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.009247782, 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000058162 2024-04-23T18:27:26.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "msgr_send_encrypted_bytes": 184512 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.640 INFO:teuthology.orchestra.run.smithi026.stdout: "cct": { 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "total_workers": 0, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "unhealthy_workers": 0 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool": { 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:26.641 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:26.642 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:26.643 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:26.644 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_items": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_bytes": 150025, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_anon_items": 32, 2024-04-23T18:27:26.645 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bytes": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_items": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:26.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_items": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pgmap_items": 0, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_co_items": 2, 2024-04-23T18:27:26.647 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_1_items": 0, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "unittest_2_items": 0 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "val": 0, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "max": 104857600, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "get_started": 0, 2024-04-23T18:27:26.648 INFO:teuthology.orchestra.run.smithi026.stdout: "get": 220, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "get_sum": 1557883, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "get_or_fail_success": 220, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "take": 0, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "take_sum": 0, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "put": 220, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "put_sum": 1557883, 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "wait": { 2024-04-23T18:27:26.649 INFO:teuthology.orchestra.run.smithi026.stdout: "avgcount": 0, 2024-04-23T18:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: "sum": 0.000000000, 2024-04-23T18:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: "avgtime": 0.000000000 2024-04-23T18:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-04-23T18:27:26.650 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-23T18:27:27.265 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-23T18:27:27.265 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-23T18:27:27.266 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-23T18:27:27.266 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-23T18:27:27.266 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-23T18:27:27.266 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:27:27.266 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:27:27.266 DEBUG:tasks.check_counter:Getting stats from b 2024-04-23T18:27:27.266 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-23T18:27:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:27 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:27:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:27 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:27:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:27 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:27:27.441 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:27 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:27:27.442 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:27 smithi110 ceph-mon[32585]: pgmap v320: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-23T18:27:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:27 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:27:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:27 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:27:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:27 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:27:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:27 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:27:27.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:27 smithi026 ceph-mon[29364]: pgmap v320: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-23T18:27:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:27 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-23T18:27:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:27 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi110 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-04-23T18:27:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:27 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi131 to 176.2M 2024-04-23T18:27:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:27 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi131 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-23T18:27:27.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:27 smithi131 ceph-mon[33041]: pgmap v320: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-23T18:27:27.516 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:27:28.856 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: pgmap v321: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.857 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:28.858 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:28.859 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:28 smithi110 ceph-mon[32585]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: pgmap v321: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:28.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:28.906 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:28 smithi026 ceph-mon[29364]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:28.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: pgmap v321: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2024-04-23T18:27:28.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:27:28.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: Adjusting osd_memory_target on smithi026 to 177.2M 2024-04-23T18:27:28.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: Unable to set osd_memory_target on smithi026 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-23T18:27:28.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:28.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:28.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:28 smithi131 ceph-mon[33041]: from='mgr.15297 ' entity='mgr.x' 2024-04-23T18:27:29.238 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-23T18:27:29.238 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:29.238 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 114305, 2024-04-23T18:27:29.238 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 81580, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 34313601, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 38314270, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 134, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 7, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 5.482048159, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 2.421164040, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 3.866142346, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.169603617, 2024-04-23T18:27:29.239 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 81583, 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2.206706745, 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000027048 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 183008, 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.022508416, 2024-04-23T18:27:29.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000122 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 45216, 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 1424 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 10192, 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 9604, 2024-04-23T18:27:29.241 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3579907, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 18265403, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 11, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.681906038, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.222427537, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.256682506, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.209426594, 2024-04-23T18:27:29.242 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9604, 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.454962204, 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000151495 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7364, 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001117814, 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000151 2024-04-23T18:27:29.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 42864, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 442816 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 62304, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 86731, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 20613648, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 98674345, 2024-04-23T18:27:29.244 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 27, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 9, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 3.285701223, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 1.407311877, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 1.999270716, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.201768776, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 86723, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2.338900149, 2024-04-23T18:27:29.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000026969 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 126067, 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.017673054, 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000140 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 2107488, 2024-04-23T18:27:29.246 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 254352 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:29.247 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 15749, 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 9.528346754, 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000605012 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 8408, 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.468653428, 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000055738 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 38932, 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 33212, 2024-04-23T18:27:29.249 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 33212, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 118.891749707, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.003579782 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 5720, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 2, 2024-04-23T18:27:29.250 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 432, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 2, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 1, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 257, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 40, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 203, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 14, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 134, 2024-04-23T18:27:29.251 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 392, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 21, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 21, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 14, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 70782, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 27627, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 5710, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 34, 2024-04-23T18:27:29.252 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 86, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 20715, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 16, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 286, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 11, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 316, 2024-04-23T18:27:29.253 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 72, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 21897020, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 6, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:29.254 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 3436, 2024-04-23T18:27:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 22631, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 10858, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 3217, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 2206, 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 31142 2024-04-23T18:27:29.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 79, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 404, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 208, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 426, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 96, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:29.257 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 4761, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 4761, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:29.258 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 11, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 409, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 16, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 3, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:29.259 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 32062, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 31987, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 31987, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 75, 2024-04-23T18:27:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 323, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 68, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 67, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 67, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 1, 2024-04-23T18:27:29.261 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 1, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 1, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 46720694, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 46899320, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 0, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7682, 2024-04-23T18:27:29.262 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 15.474758917, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002014417 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 0 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 178, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 5213, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 5035, 2024-04-23T18:27:29.263 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 48, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 272, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 224, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 257, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 9289, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 9032, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 21, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 5898, 2024-04-23T18:27:29.264 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 5877, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 69620, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 231800 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 63798, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 82, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 38932, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 289, 2024-04-23T18:27:29.265 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 9950, 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 4762, 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 45.150217155, 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.009481355 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-23T18:27:29.266 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 145, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.049942232, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000344429 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.267 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 6829, 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 6.306581346, 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000923499 2024-04-23T18:27:29.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.269 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:29.270 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.271 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.272 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-23T18:27:29.272 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 10, 2024-04-23T18:27:29.272 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001426615, 2024-04-23T18:27:29.272 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000142661 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7, 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.010531033, 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001504433 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.273 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 10, 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.609125596, 2024-04-23T18:27:29.274 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.060912559 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1, 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000215072, 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000215072 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 16589, 2024-04-23T18:27:29.275 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 8.084047020, 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000487313 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 86, 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.162718019, 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001892069 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.276 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 4, 2024-04-23T18:27:29.278 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.844756842, 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.461189210 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-23T18:27:29.279 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000838740, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000838740 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.280 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.281 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7, 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.017237757, 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002462536 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-23T18:27:29.282 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 4761, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 56.654112280, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.011899624 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.283 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-23T18:27:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 5, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 3, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 2, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 4775, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 2387, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 591, 2024-04-23T18:27:29.285 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:29.286 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:29.287 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:29.288 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-23T18:27:29.289 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 283831, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 225, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-23T18:27:29.290 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:29.291 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 729361, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 5529, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-23T18:27:29.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 17333, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 47101228, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 37, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 17333, 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-23T18:27:29.293 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 17333, 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 86.033627955, 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.004963573 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 17333, 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 23676 2024-04-23T18:27:29.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 17333, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 338, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 16995, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 391, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 2446, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 336, 2024-04-23T18:27:29.295 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 8018, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 372, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 2, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 5301, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-23T18:27:29.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 1, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 4892, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-23T18:27:29.297 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-23T18:27:29.298 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-23T18:27:29.299 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 98, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 13, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 12, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 12, 2024-04-23T18:27:29.300 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 0, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 1495, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 4, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 418 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 1, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:29.301 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 498 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:29.302 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 10, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 4761, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 4761, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:29.303 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 186797, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 41630688, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 186797, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 186797, 2024-04-23T18:27:29.304 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 41630688, 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:29.305 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 17333, 2024-04-23T18:27:29.306 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 47582977, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 17333, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 47582977, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.307 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:29.308 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 17333, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 17333, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 17333, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 17333, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 4761, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 481781, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:29.310 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 4761, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 336, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 481781, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.311 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x55a8e3946500": { 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 32062, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 42705016, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:29.312 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 32062, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 7682, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 42705016, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:29.313 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:29.314 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.314 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:29.314 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds.exported=16 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds.imported=11 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=79 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=404 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-23T18:27:30.298 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-23T18:27:30.299 DEBUG:tasks.check_counter:Getting stats from e 2024-04-23T18:27:30.299 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-23T18:27:30.540 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:27:30.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:30 smithi110 ceph-mon[32585]: pgmap v322: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:27:30.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:30 smithi026 ceph-mon[29364]: pgmap v322: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:27:30.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:30 smithi131 ceph-mon[33041]: pgmap v322: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2024-04-23T18:27:32.192 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-23T18:27:32.192 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:32.192 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 1642, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 1643, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 80172263, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 521845, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 8, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 4, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.169367380, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.042669482, 2024-04-23T18:27:32.193 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.075662715, 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.039116801, 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1641, 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.048367771, 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000029474 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:32.194 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 755, 2024-04-23T18:27:32.195 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 716, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 77352872, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 286726, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 6, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.109292871, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.021302138, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.080524718, 2024-04-23T18:27:32.196 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.015959047, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 715, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.021528908, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000030110 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.197 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1730512, 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 130208 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 555, 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 710, 2024-04-23T18:27:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 80153605, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 616822, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 8, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 6, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.080850215, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.014821071, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.049742421, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.011939203, 2024-04-23T18:27:32.199 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 708, 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.024415804, 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000034485 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.200 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 442240 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 4511, 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.094697317, 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000020992 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.202 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-23T18:27:32.203 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 0, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 0, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-23T18:27:32.204 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 7014, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 47, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 6967, 2024-04-23T18:27:32.205 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 6947, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 15, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 0, 2024-04-23T18:27:32.206 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 0, 2024-04-23T18:27:32.207 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 0, 2024-04-23T18:27:32.208 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:32.209 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:27:32.210 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 0 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 0, 2024-04-23T18:27:32.211 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:32.212 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-23T18:27:32.213 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 0, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-04-23T18:27:32.214 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 48546, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 79, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 5, 2024-04-23T18:27:32.215 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 239385875, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 239365826, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.216 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 48546 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 222, 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 7236, 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 7014, 2024-04-23T18:27:32.217 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 44, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 55, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 11, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 7014, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 7334, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 320, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 0, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 0, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 0, 2024-04-23T18:27:32.218 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 63132, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 215416 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 0, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 0, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 0, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 0, 2024-04-23T18:27:32.219 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 0, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.220 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-23T18:27:32.221 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.223 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:32.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.226 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-23T18:27:32.226 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.226 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.226 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.227 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.229 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:32.230 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-23T18:27:32.231 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.232 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.233 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:32.234 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.235 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:32.237 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-23T18:27:32.238 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 2, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 4, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 0, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 0, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 2906, 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:32.240 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:32.241 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:32.242 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-23T18:27:32.243 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 253203, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 424, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-23T18:27:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:32.245 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 5708364, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 35618, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-23T18:27:32.246 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 2682, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 21, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 3, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 2682, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2682, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2.240964332, 2024-04-23T18:27:32.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000835557 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2682, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2683 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 2682, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 2682, 2024-04-23T18:27:32.248 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 1484, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 1196, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-23T18:27:32.249 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 1, 2024-04-23T18:27:32.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:32.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-23T18:27:32.252 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-23T18:27:32.253 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 98, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 12, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 11, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 11, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 0, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 0, 2024-04-23T18:27:32.254 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 2, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 0 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 0, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 0, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 0, 2024-04-23T18:27:32.256 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 0, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 2948, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 237397082, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:32.257 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 2948, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 2948, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 237397082, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.258 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:32.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 2682, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 235199666, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 2682, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 235199666, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.260 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:32.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 2682, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 2682, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 2682, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 2682, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.262 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:32.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.264 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x557a5d11a6e0": { 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:32.265 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:32.266 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:32.267 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:32.267 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.267 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:32.267 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-23T18:27:32.901 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:32 smithi110 ceph-mon[32585]: pgmap v323: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-23T18:27:32.903 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-23T18:27:32.903 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-23T18:27:32.903 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-23T18:27:32.903 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-23T18:27:32.904 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-23T18:27:32.904 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:27:32.904 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:27:32.904 DEBUG:tasks.check_counter:Getting stats from h 2024-04-23T18:27:32.904 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-23T18:27:32.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:32 smithi026 ceph-mon[29364]: pgmap v323: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-23T18:27:32.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:32 smithi131 ceph-mon[33041]: pgmap v323: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2024-04-23T18:27:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:27:34.898 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-23T18:27:34.898 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:34.898 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 195647, 2024-04-23T18:27:34.898 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 257280, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 52029921, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 232713345, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 13, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 7, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 10.148042274, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 4.714330740, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 6.296261268, 2024-04-23T18:27:34.899 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.303826317, 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 257276, 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 3.819058542, 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000014844 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 402756, 2024-04-23T18:27:34.900 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.048363404, 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000120 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 103830, 2024-04-23T18:27:34.901 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 124203, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 25376346, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 95953645, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 13, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 5.674339276, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 2.354260896, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 3.698931699, 2024-04-23T18:27:34.902 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.241293819, 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 124205, 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.985099792, 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000015982 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:34.903 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 207410, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.028161816, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000135 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1746720, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 139872 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 13080, 2024-04-23T18:27:34.904 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 13265, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3963871, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 98342482, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 11, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 7, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.933303184, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.366839591, 2024-04-23T18:27:34.905 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.200859699, 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.301845589, 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 13262, 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.373719332, 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000028179 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.906 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 77, 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000014380, 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000186 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 6800, 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 456576 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.907 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 54583, 2024-04-23T18:27:34.908 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 30.782241535, 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000563952 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 11375, 2024-04-23T18:27:34.909 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.907544021, 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000079784 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 75557, 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 68502, 2024-04-23T18:27:34.910 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 68502, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 228.586855792, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.003336936 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 7055, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 2, 2024-04-23T18:27:34.911 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 29, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 2, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 1, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 322, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 156, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 150, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 16, 2024-04-23T18:27:34.912 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 95, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 454, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 30, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 30, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 18, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 122560, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 85389, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 7043, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 11, 2024-04-23T18:27:34.913 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 785, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 2220, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 13, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 350, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 9, 2024-04-23T18:27:34.914 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 196, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 4, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 234, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 0, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:34.915 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 15, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 17824, 2024-04-23T18:27:34.916 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 22163, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 27686, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 4714, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 2782, 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 36413 2024-04-23T18:27:34.917 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 36, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 8, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 5, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 15, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 344, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:34.918 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 6774, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 6774, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:34.919 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 11, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 565, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 14, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 3, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:34.920 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 48547, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 48547, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-23T18:27:34.921 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 79, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 79, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 5, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-04-23T18:27:34.922 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 239392199, 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 0, 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 27355, 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 48.050910582, 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001756567 2024-04-23T18:27:34.923 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 0 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 228, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 7284, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 7056, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 49, 2024-04-23T18:27:34.924 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 64, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 15, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 322, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 12354, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 12032, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 30, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 7617, 2024-04-23T18:27:34.925 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 7587, 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 75360, 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 223608 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 112472, 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 26987, 2024-04-23T18:27:34.926 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 75557, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 1272, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 54018, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 6844, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 35.617089519, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.005204133 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.927 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 16458, 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 84.959115203, 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.005162177 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 17511, 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 17.276597901, 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000986614 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.930 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:34.931 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.931 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.931 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.931 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.931 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.934 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:34.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.936 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.936 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:34 smithi110 ceph-mon[32585]: pgmap v324: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.6 KiB/s wr, 6 op/s 2024-04-23T18:27:34.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.937 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-23T18:27:34.937 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.937 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.937 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 13, 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.013255252, 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001019634 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-23T18:27:34.938 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.939 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-23T18:27:34.940 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 18101, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 28.167979311, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001556155 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2182, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 30.681486979, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.014061176 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.941 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.942 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.943 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.006924042, 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002308014 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:34.945 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 616, 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.376961067, 2024-04-23T18:27:34.946 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000611949 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.947 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.948 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 6774, 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 31.487446518, 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.004648279 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.949 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.950 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 2, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 2, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 7388, 2024-04-23T18:27:34.951 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 3694, 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 599, 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 120, 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 120, 2024-04-23T18:27:34.952 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:34.953 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:34.954 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:34.955 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:34.956 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 312774, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 239, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-23T18:27:34.957 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:34.958 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 1052483, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 9853, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-23T18:27:34.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 35664, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 236854913, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 28, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 35664, 2024-04-23T18:27:34.960 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 35664, 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 70.191880397, 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001968143 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 35664, 2024-04-23T18:27:34.961 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 36268 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 35664, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 409, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 35255, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-23T18:27:34.962 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 0, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 10, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 407, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 27762, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 241, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-23T18:27:34.963 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 2, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 6850, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 20, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:34.964 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-23T18:27:34.965 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-23T18:27:34.966 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-23T18:27:34.967 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 98, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 12, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 12, 2024-04-23T18:27:34.968 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 12, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 0, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 713, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 4, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 259 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.969 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 1, 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 890 2024-04-23T18:27:34.970 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 10, 2024-04-23T18:27:34.971 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 6774, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 6774, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:34.972 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 312553, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 53221275, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 312553, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 312553, 2024-04-23T18:27:34.973 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 53221275, 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:34.974 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 35664, 2024-04-23T18:27:34.975 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 237540671, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 35664, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 237540671, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:34.976 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 35664, 2024-04-23T18:27:34.977 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 35664, 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 35664, 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 35664, 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.978 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 6774, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 685790, 2024-04-23T18:27:34.979 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 6774, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 407, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 685790, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x561d14676500": { 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-23T18:27:34.981 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 48547, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 235197895, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 48547, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 27355, 2024-04-23T18:27:34.982 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 235197895, 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:34.983 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-23T18:27:34.984 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:34 smithi026 ceph-mon[29364]: pgmap v324: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.6 KiB/s wr, 6 op/s 2024-04-23T18:27:34.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:34 smithi131 ceph-mon[33041]: pgmap v324: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.6 KiB/s wr, 6 op/s 2024-04-23T18:27:35.745 INFO:tasks.check_counter:Daemon mds.h mds.exported=13 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds.imported=9 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=36 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=8 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:27:35.746 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:27:35.746 DEBUG:tasks.check_counter:Getting stats from k 2024-04-23T18:27:35.747 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-23T18:27:35.983 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:27:36.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:36 smithi110 ceph-mon[32585]: pgmap v325: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 8 op/s 2024-04-23T18:27:36.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:36 smithi026 ceph-mon[29364]: pgmap v325: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 8 op/s 2024-04-23T18:27:36.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:36 smithi131 ceph-mon[33041]: pgmap v325: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 8 op/s 2024-04-23T18:27:37.578 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 0, 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 2, 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 0, 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:37.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.000960141, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.000102614, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:37.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:37.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 226, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 213, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1592688, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 130272, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:37.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.041857641, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.011856275, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.035320547, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.004811055, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 213, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.010337094, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000048530 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1592688, 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 130272 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 2, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 161, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 320, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 184416, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.009998242, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.006585554, 2024-04-23T18:27:37.585 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.001447680, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 158, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.015149575, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000095883 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:37.586 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 184416 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 0, 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-23T18:27:37.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:37.588 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:37.590 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-23T18:27:37.591 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 148933, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 30, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-23T18:27:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 2, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-23T18:27:37.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 224, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 1558403, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 224, 2024-04-23T18:27:37.595 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 224, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 1558403, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:37.596 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-23T18:27:37.597 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-23T18:27:38.268 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-23T18:27:38.268 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-23T18:27:38.268 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-23T18:27:38.268 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-23T18:27:38.269 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-23T18:27:38.269 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:27:38.269 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:27:38.269 DEBUG:tasks.check_counter:Getting stats from c 2024-04-23T18:27:38.269 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-23T18:27:38.520 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:27:38.924 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:38 smithi131 ceph-mon[33041]: pgmap v326: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:38.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:38 smithi110 ceph-mon[32585]: pgmap v326: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:38.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:38 smithi026 ceph-mon[29364]: pgmap v326: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 1371, 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 1346, 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 14054281, 2024-04-23T18:27:40.218 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 470814, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 6, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.133180810, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.037537124, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.047633828, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.039231484, 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:40.219 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1345, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.045457078, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000033797 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.220 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 443472, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 95888 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 338, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 453, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 12669222, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 397431, 2024-04-23T18:27:40.221 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 4, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.036725593, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.012071706, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.014335839, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.007281301, 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:40.222 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 451, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.016563471, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000036726 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.223 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 290256 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 552, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 552, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 16919862, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 174776, 2024-04-23T18:27:40.224 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 4, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.048479009, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.013241897, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.018295085, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.013450398, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 551, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.019215374, 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000034873 2024-04-23T18:27:40.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:27:40.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:40.227 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3504, 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.122518159, 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000034965 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:40.228 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 0, 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 0, 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-04-23T18:27:40.229 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:40.230 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 0, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 4990, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 3423, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 1567, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 101, 2024-04-23T18:27:40.231 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 7, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:40.232 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:40.233 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 66, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 21133856, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 6, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:40.234 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:27:40.235 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 0, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 0 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-04-23T18:27:40.236 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:40.237 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-04-23T18:27:40.238 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-04-23T18:27:40.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 32017, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-04-23T18:27:40.240 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 0, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 2, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 40, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 46720694, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 46916883, 2024-04-23T18:27:40.241 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 46908576, 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 32017 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.242 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 176, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 5167, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 4991, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 25, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 49, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 24, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 4990, 2024-04-23T18:27:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 5266, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 276, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 0, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 0, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 0, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 56648, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 215416 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-04-23T18:27:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-04-23T18:27:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.249 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:40.250 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.252 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-04-23T18:27:40.252 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.252 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.253 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.254 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-04-23T18:27:40.255 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.257 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:40.258 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.260 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.261 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.262 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:40.263 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 2, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 4, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 2, 2024-04-23T18:27:40.264 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 0, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 0, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 0, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 3009, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-04-23T18:27:40.265 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:40.266 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:40.267 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:40.268 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:40.269 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 212304, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 222, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:40.270 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 167, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:40.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 3970661, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 23567, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-04-23T18:27:40.272 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 2064, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 26, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 4, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 2064, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2064, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 1.067976637, 2024-04-23T18:27:40.273 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000517430 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2064, 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 2065 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 2064, 2024-04-23T18:27:40.274 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 2064, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 0, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 1118, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 0, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 943, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 0, 2024-04-23T18:27:40.275 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:40.276 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 2, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:40.277 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-04-23T18:27:40.278 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-04-23T18:27:40.279 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 98, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 8, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 7, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 7, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-04-23T18:27:40.280 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 0, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 2, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 0 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 0, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-04-23T18:27:40.281 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-04-23T18:27:40.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 0, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 0, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 0, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:40.283 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 2257, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 43427517, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 2257, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2257, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 43427517, 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:40.284 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:40.285 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2064, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 42625413, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2064, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 42625413, 2024-04-23T18:27:40.286 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-04-23T18:27:40.287 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2064, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 2064, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2064, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2064, 2024-04-23T18:27:40.288 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:40.289 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:40.290 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x5646efb5e6e0": { 2024-04-23T18:27:40.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:40.292 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:40.293 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-04-23T18:27:40.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:40 smithi110 ceph-mon[32585]: pgmap v327: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:40.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:40 smithi026 ceph-mon[29364]: pgmap v327: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:40.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:40 smithi131 ceph-mon[33041]: pgmap v327: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.3 KiB/s wr, 6 op/s 2024-04-23T18:27:41.009 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-23T18:27:41.009 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-23T18:27:41.009 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-23T18:27:41.010 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-23T18:27:41.010 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-23T18:27:41.010 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:27:41.010 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:27:41.010 DEBUG:tasks.check_counter:Getting stats from f 2024-04-23T18:27:41.010 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-23T18:27:41.250 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:27:42.884 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:42 smithi131 ceph-mon[33041]: pgmap v328: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-23T18:27:42.911 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 634, 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 636, 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 25337030, 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 201676, 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 7, 2024-04-23T18:27:42.912 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.065806340, 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.014723447, 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.029723447, 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.015768572, 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:42.913 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 634, 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.020609128, 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000032506 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 757, 2024-04-23T18:27:42.915 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 913, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 33750716, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 682855, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 8, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 6, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.078894734, 2024-04-23T18:27:42.916 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.023173588, 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.029696357, 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.019555077, 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 911, 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.038254279, 2024-04-23T18:27:42.917 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000041991 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T18:27:42.918 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 444032 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 1413, 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 1374, 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 23798956, 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 496552, 2024-04-23T18:27:42.919 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 4, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.161307452, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.044748333, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.076843190, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.040427926, 2024-04-23T18:27:42.920 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1373, 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.053890263, 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000039250 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.921 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1731056, 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 131488 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-04-23T18:27:42.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:42.923 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 4340, 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.105842663, 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000024387 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:42.924 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-04-23T18:27:42.925 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 0, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 0, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-04-23T18:27:42.926 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 7956, 2024-04-23T18:27:42.927 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 46, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 7910, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 7890, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 0, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 0, 2024-04-23T18:27:42.928 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 12, 2024-04-23T18:27:42.929 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 0, 2024-04-23T18:27:42.929 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-04-23T18:27:42.930 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:42.932 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:42.933 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 0, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 0, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 0 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-04-23T18:27:42.934 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 0, 2024-04-23T18:27:42.935 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:27:42.936 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-04-23T18:27:42.937 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 48699, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-04-23T18:27:42.938 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 61, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 5, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 84629478, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 84627723, 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-04-23T18:27:42.939 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 48699 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 274, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 7940, 2024-04-23T18:27:42.940 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 7666, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 30, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 33, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 3, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 7956, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 8028, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 72, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 0, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 0, 2024-04-23T18:27:42.941 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 0, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 62236, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 215416 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 0, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 0, 2024-04-23T18:27:42.942 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 0, 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-04-23T18:27:42.943 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.944 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.946 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:42.947 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.948 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.949 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-04-23T18:27:42.950 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:42 smithi110 ceph-mon[32585]: pgmap v328: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-23T18:27:42.950 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.950 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.957 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.958 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.960 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 2, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 5, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 3, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 4, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-04-23T18:27:42.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 0, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 0, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 2941, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:42.963 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:42.965 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:42.966 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:42.967 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 233796, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 336, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:42.968 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 167, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 6363292, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 37962, 2024-04-23T18:27:42.969 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-04-23T18:27:42.970 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 2533, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 21, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 3, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 2533, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2533, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 1.448968374, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000572036 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.971 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2533, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 2534 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 2533, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 2533, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 0, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-04-23T18:27:42.972 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 1404, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 1127, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-04-23T18:27:42.973 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 1, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 0, 2024-04-23T18:27:42.974 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:42.975 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-04-23T18:27:42.976 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 98, 2024-04-23T18:27:42.977 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 12, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 9, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 9, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 0, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 2, 2024-04-23T18:27:42.978 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 0 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 0, 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-04-23T18:27:42.979 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 0, 2024-04-23T18:27:42.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 2800, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 82618260, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:42.981 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 2800, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2800, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 82618260, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.982 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:42.983 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2533, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 80446130, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2533, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 80446130, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.984 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:42.985 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2533, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 2533, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2533, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2533, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.986 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:42.987 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:42.988 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x55d3a2d566e0": { 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-04-23T18:27:42.990 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:42.991 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-04-23T18:27:42.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:42 smithi026 ceph-mon[29364]: pgmap v328: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-23T18:27:43.673 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-23T18:27:43.673 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-23T18:27:43.674 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-23T18:27:43.674 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-23T18:27:43.674 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-23T18:27:43.674 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:27:43.674 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:27:43.674 DEBUG:tasks.check_counter:Getting stats from i 2024-04-23T18:27:43.674 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-23T18:27:43.911 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:27:44.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:44 smithi110 ceph-mon[32585]: pgmap v329: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-23T18:27:44.945 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:44 smithi131 ceph-mon[33041]: pgmap v329: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-23T18:27:44.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:44 smithi026 ceph-mon[29364]: pgmap v329: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-04-23T18:27:45.484 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 12678, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 13349, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 3319246, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 46097930, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 13, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 6, 2024-04-23T18:27:45.485 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.846242909, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.307489528, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.281747858, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.230630269, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 13348, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.331798105, 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000024857 2024-04-23T18:27:45.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 7307, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.001097110, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000150 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 4560, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 5600 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.487 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 122970, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 102523, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 37685598, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 54216844, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 10, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 7, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 5.581958554, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 2.284814399, 2024-04-23T18:27:45.488 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 3.968737307, 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.217435877, 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 102522, 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 1.558611528, 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000015202 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:45.489 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 206907, 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.025459744, 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000123 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1739104, 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 137648 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:45.490 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 84226, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 126348, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 26212470, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 94341346, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 15, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 6, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 4.532880425, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 1.988261290, 2024-04-23T18:27:45.491 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 2.883515145, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.149039805, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 126345, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 2.235511584, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000017693 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 183633, 2024-04-23T18:27:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.025118218, 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000136 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 445792 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-04-23T18:27:45.493 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 27453, 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 14.382374165, 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000523890 2024-04-23T18:27:45.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 12989, 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.790312268, 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000060844 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 55635, 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 49745, 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 49745, 2024-04-23T18:27:45.496 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 127.588218931, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002564845 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 5890, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 11, 2024-04-23T18:27:45.497 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 328, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 51, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 222, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 55, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 256, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 104, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 15, 2024-04-23T18:27:45.498 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 15, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 16, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 117309, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 53990, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 5888, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 26, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-04-23T18:27:45.499 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 1752, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 1415, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 3, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 81, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 12, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 205, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:27:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 4, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 234, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:27:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 2, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 4295, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 32707, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:27:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 15573, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 7341, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 2693, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 48726 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 177, 2024-04-23T18:27:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 202, 2024-04-23T18:27:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 109, 2024-04-23T18:27:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 201, 2024-04-23T18:27:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 202, 2024-04-23T18:27:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-04-23T18:27:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 7637, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 7637, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:27:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_path": 11, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_quiesce_inode": 403, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 4, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-04-23T18:27:45.506 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 48702, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-04-23T18:27:45.507 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 48702, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 61, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 61, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 5, 2024-04-23T18:27:45.508 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 84631200, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 0, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 13883, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 24.511634970, 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.001765586 2024-04-23T18:27:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 0 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 281, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 7966, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 7685, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 46, 2024-04-23T18:27:45.510 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 134, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 88, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 328, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 11603, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 11275, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 15, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 7903, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 7888, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 70236, 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 223608 2024-04-23T18:27:45.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 99121, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 33, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 55635, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 1245, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 17311, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 7632, 2024-04-23T18:27:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 70.269018734, 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.009207156 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 187, 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.130270421, 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000696633 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.513 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.514 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 11960, 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 10.864572837, 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000908409 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-04-23T18:27:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2, 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000366105, 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000183052 2024-04-23T18:27:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-04-23T18:27:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.521 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 20555, 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 8.353201034, 2024-04-23T18:27:45.522 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000406382 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1749, 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 25.223325494, 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.014421569 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-04-23T18:27:45.523 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.525 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-04-23T18:27:45.526 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 23, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.056555676, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002458942 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.527 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.528 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.529 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 7637, 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 12.690908630, 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.001661766 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.530 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 2, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 5, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 3, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 2, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 6039, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 3019, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 607, 2024-04-23T18:27:45.532 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:45.533 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:45.534 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:45.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 303346, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 302, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 88, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 1, 2024-04-23T18:27:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8032, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 167, 2024-04-23T18:27:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 1043404, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 8038, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-04-23T18:27:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 22911, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 81975607, 2024-04-23T18:27:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 33, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 22911, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 22911, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 48.049949561, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002097243 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-04-23T18:27:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 22911, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 23458 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 22911, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 386, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 22525, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-04-23T18:27:45.542 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 0, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 10, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 386, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 14269, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 220, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-04-23T18:27:45.543 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 7668, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 0, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 20, 2024-04-23T18:27:45.544 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-04-23T18:27:45.545 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-04-23T18:27:45.546 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-04-23T18:27:45.547 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 98, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 12, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 12, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 12, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-04-23T18:27:45.548 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 653, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 0, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 232 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 1, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:27:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 449 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-04-23T18:27:45.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 10, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 7637, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 7637, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:45.551 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 219870, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 47410212, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 219870, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:45.552 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 219870, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 47410212, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:45.554 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 22911, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 82748152, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 22911, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 82748152, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.555 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-04-23T18:27:45.556 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 22911, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 22911, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 22911, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 22911, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.557 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 7637, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 772545, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 7637, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 386, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 772545, 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:45.559 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x560652862500": { 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 1722, 2024-04-23T18:27:45.560 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 48702, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 80436896, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 48702, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 13883, 2024-04-23T18:27:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 80435174, 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:45.562 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-04-23T18:27:46.325 INFO:tasks.check_counter:Daemon mds.i mds.exported=3 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds.imported=12 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=177 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=202 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:27:46.326 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:27:46.326 DEBUG:tasks.check_counter:Getting stats from l 2024-04-23T18:27:46.327 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 2a701120-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-23T18:27:46.567 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:27:46.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:46 smithi110 ceph-mon[32585]: pgmap v330: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:27:46.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:46 smithi026 ceph-mon[29364]: pgmap v330: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:27:46.973 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:46 smithi131 ceph-mon[33041]: pgmap v330: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:27:48.160 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 4, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 4, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 448, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 4, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 0, 2024-04-23T18:27:48.161 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.001292512, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.000037571, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.000576657, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000749764, 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000249921 2024-04-23T18:27:48.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.163 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 2, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 164, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 320, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 184944, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 5, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 3, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.010226577, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.006650087, 2024-04-23T18:27:48.164 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.001397495, 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 161, 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.017824824, 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000110713 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:48.165 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 184944 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:27:48.166 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 226, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 215, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 1590848, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 131904, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 2, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 1, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.042212727, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.012578578, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.034902883, 2024-04-23T18:27:48.167 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.004666767, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 214, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.009718545, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000045413 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:48.168 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1590848, 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 131904 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 0, 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-04-23T18:27:48.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:27:48.170 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:27:48.171 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:27:48.172 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:27:48.173 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 149017, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 31, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-04-23T18:27:48.174 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:27:48.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 928, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 2, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-04-23T18:27:48.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 228, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 1558987, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 228, 2024-04-23T18:27:48.177 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 228, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1558987, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-04-23T18:27:48.178 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:48.179 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-04-23T18:27:48.179 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-04-23T18:27:48.741 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:48 smithi131 ceph-mon[33041]: pgmap v331: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:27:48.812 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:27:48.813 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T18:27:48.824 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T18:27:48.824 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:27:48.824 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:27:48.846 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-23T18:27:48.846 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:27:48.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:27:48.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:48 smithi110 ceph-mon[32585]: pgmap v331: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:48.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:48 smithi026 ceph-mon[29364]: pgmap v331: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:49.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.069+0000 7f1112108640 1 -- 172.21.15.26:0/3956136080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101ac0 msgr2=0x7f110c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.069+0000 7f1112108640 1 --2- 172.21.15.26:0/3956136080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101ac0 0x7f110c105ab0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1100009900 tx=0x7f110002a3f0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 -- 172.21.15.26:0/3956136080 shutdown_connections 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 --2- 172.21.15.26:0/3956136080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f110c106190 0x7f110c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 --2- 172.21.15.26:0/3956136080 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101ac0 0x7f110c105ab0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 --2- 172.21.15.26:0/3956136080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c101110 0x7f110c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 -- 172.21.15.26:0/3956136080 >> 172.21.15.26:0/3956136080 conn(0x7f110c077fb0 msgr2=0x7f110c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 -- 172.21.15.26:0/3956136080 shutdown_connections 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 -- 172.21.15.26:0/3956136080 wait complete. 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 Processor -- start 2024-04-23T18:27:49.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.070+0000 7f1112108640 1 -- start start 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f110c106190 0x7f110c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c19cf00 0x7f110c19d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1110b0 con 0x7f110c101110 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f110c110f30 con 0x7f110c106190 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f110c111230 con 0x7f110c19cf00 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:32940/0 (socket says 172.21.15.26:32940) 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 -- 172.21.15.26:0/710716521 learned_addr learned my addr 172.21.15.26:0/710716521 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110bfff640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c19cf00 0x7f110c19d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 -- 172.21.15.26:0/710716521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c19cf00 msgr2=0x7f110c19d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c19cf00 0x7f110c19d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 -- 172.21.15.26:0/710716521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f110c106190 msgr2=0x7f110c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f110c106190 0x7f110c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 -- 172.21.15.26:0/710716521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c19d9f0 con 0x7f110c101110 2024-04-23T18:27:49.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f110b7fe640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00c920 tx=0x7f10fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc007c70 con 0x7f110c101110 2024-04-23T18:27:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c19dce0 con 0x7f110c101110 2024-04-23T18:27:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10fc007e10 con 0x7f110c101110 2024-04-23T18:27:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc018820 con 0x7f110c101110 2024-04-23T18:27:49.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.071+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c1a54d0 con 0x7f110c101110 2024-04-23T18:27:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.072+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f110c0691c0 con 0x7f110c101110 2024-04-23T18:27:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.074+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10fc00f450 con 0x7f110c101110 2024-04-23T18:27:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.074+0000 7f1108ff9640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 0x7f10f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.074+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f10fc0d0920 con 0x7f110c101110 2024-04-23T18:27:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.074+0000 7f110affd640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 0x7f10f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.075+0000 7f110affd640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 0x7f10f40a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11000096c0 tx=0x7f1100031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.079+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10fc014030 con 0x7f110c101110 2024-04-23T18:27:49.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.269+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f110c101b50 con 0x7f110c101110 2024-04-23T18:27:49.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.271+0000 7f1108ff9640 1 -- 172.21.15.26:0/710716521 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+23127 (secure 0 0 0) 0x7f10fc09cfc0 con 0x7f110c101110 2024-04-23T18:27:49.264 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:49.264 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":98,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:27:16.798801+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:27:49.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 msgr2=0x7f10f40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 0x7f10f40a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11000096c0 tx=0x7f1100031400 comp rx=0 tx=0).stop 2024-04-23T18:27:49.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 msgr2=0x7f110c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00c920 tx=0x7f10fc00cde0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 shutdown_connections 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f10f40a3d20 0x7f10f40a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f110c106190 0x7f110c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f110c19cf00 0x7f110c19d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 --2- 172.21.15.26:0/710716521 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f110c101110 0x7f110c198690 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 >> 172.21.15.26:0/710716521 conn(0x7f110c077fb0 msgr2=0x7f110c104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 shutdown_connections 2024-04-23T18:27:49.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.273+0000 7f1112108640 1 -- 172.21.15.26:0/710716521 wait complete. 2024-04-23T18:27:49.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:10:34.851766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:15:03.735783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:15:04.991252+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:27:49.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:27:49.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1857668621 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b3810c510 msgr2=0x7f5b3810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1857668621 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b3810c510 0x7f5b3810e9b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009900 tx=0x7f5b3002a510 comp rx=0 tx=0).stop 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1857668621 shutdown_connections 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1857668621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b38103af0 0x7f5b3810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1857668621 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b3810c510 0x7f5b3810e9b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1857668621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b381031d0 0x7f5b381035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1857668621 >> 172.21.15.26:0/1857668621 conn(0x7f5b38077f50 msgr2=0x7f5b38078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1857668621 shutdown_connections 2024-04-23T18:27:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.431+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1857668621 wait complete. 2024-04-23T18:27:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 Processor -- start 2024-04-23T18:27:49.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 -- start start 2024-04-23T18:27:49.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b381031d0 0x7f5b38100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b3810c510 0x7f5b381018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38112200 con 0x7f5b38103af0 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38112080 con 0x7f5b381031d0 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.432+0000 7f5b3d56f640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38112380 con 0x7f5b3810c510 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:32958/0 (socket says 172.21.15.26:32958) 2024-04-23T18:27:49.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 -- 172.21.15.26:0/1268410200 learned_addr learned my addr 172.21.15.26:0/1268410200 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b36ffd640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b381031d0 0x7f5b38100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b377fe640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b3810c510 0x7f5b381018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 -- 172.21.15.26:0/1268410200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b3810c510 msgr2=0x7f5b381018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b3810c510 0x7f5b381018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 -- 172.21.15.26:0/1268410200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b381031d0 msgr2=0x7f5b38100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b381031d0 0x7f5b38100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 -- 172.21.15.26:0/1268410200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b381a6c60 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b367fc640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00d950 tx=0x7f5b2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c017070 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b2c009c30 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b381a6f50 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00f620 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.433+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b381a7410 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.434+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b38068d70 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.435+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b2c005030 con 0x7f5b38103af0 2024-04-23T18:27:49.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.436+0000 7f5b1ffff640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 0x7f5b200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.436+0000 7f5b36ffd640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 0x7f5b200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.436+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f5b2c0d04c0 con 0x7f5b38103af0 2024-04-23T18:27:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.437+0000 7f5b36ffd640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 0x7f5b200a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009f20 tx=0x7f5b280094f0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.440+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b2c09cb60 con 0x7f5b38103af0 2024-04-23T18:27:49.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.625+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b38107690 con 0x7f5b38103af0 2024-04-23T18:27:49.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.626+0000 7f5b1ffff640 1 -- 172.21.15.26:0/1268410200 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f5b2c009de0 con 0x7f5b38103af0 2024-04-23T18:27:49.619 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-04-23T18:27:49.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 msgr2=0x7f5b200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 0x7f5b200a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009f20 tx=0x7f5b280094f0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 msgr2=0x7f5b38101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00d950 tx=0x7f5b2c00de10 comp rx=0 tx=0).stop 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 shutdown_connections 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f5b200a3d20 0x7f5b200a61e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b3810c510 0x7f5b381018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5b38103af0 0x7f5b38101360 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 --2- 172.21.15.26:0/1268410200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b381031d0 0x7f5b38100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 >> 172.21.15.26:0/1268410200 conn(0x7f5b38077f50 msgr2=0x7f5b380fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 shutdown_connections 2024-04-23T18:27:49.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.628+0000 7f5b3d56f640 1 -- 172.21.15.26:0/1268410200 wait complete. 2024-04-23T18:27:49.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:27:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 -- 172.21.15.26:0/2330466452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230101a30 msgr2=0x7fa230105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 --2- 172.21.15.26:0/2330466452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230101a30 0x7fa230105a70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa224009900 tx=0x7fa22402a410 comp rx=0 tx=0).stop 2024-04-23T18:27:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 -- 172.21.15.26:0/2330466452 shutdown_connections 2024-04-23T18:27:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 --2- 172.21.15.26:0/2330466452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230106150 0x7fa23010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 --2- 172.21.15.26:0/2330466452 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230101a30 0x7fa230105a70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 --2- 172.21.15.26:0/2330466452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa230101080 0x7fa230101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.744+0000 7fa2353bf640 1 -- 172.21.15.26:0/2330466452 >> 172.21.15.26:0/2330466452 conn(0x7fa230078000 msgr2=0x7fa230078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- 172.21.15.26:0/2330466452 shutdown_connections 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- 172.21.15.26:0/2330466452 wait complete. 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 Processor -- start 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- start start 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230101080 0x7fa2301986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 0x7fa23019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fa2301121d0 con 0x7fa230106150 2024-04-23T18:27:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa230112050 con 0x7fa23019cfb0 2024-04-23T18:27:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.745+0000 7fa2353bf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fa230112350 con 0x7fa230101080 2024-04-23T18:27:49.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:32980/0 (socket says 172.21.15.26:32980) 2024-04-23T18:27:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 -- 172.21.15.26:0/4282288790 learned_addr learned my addr 172.21.15.26:0/4282288790 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22effd640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230101080 0x7fa2301986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22f7fe640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 0x7fa23019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 -- 172.21.15.26:0/4282288790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230101080 msgr2=0x7fa2301986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230101080 0x7fa2301986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 -- 172.21.15.26:0/4282288790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 msgr2=0x7fa23019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 0x7fa23019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 -- 172.21.15.26:0/4282288790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23019daa0 con 0x7fa230106150 2024-04-23T18:27:49.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22f7fe640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 0x7fa23019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa22e7fc640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa224006010 tx=0x7fa224004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22403f030 con 0x7fa230106150 2024-04-23T18:27:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa224003aa0 con 0x7fa230106150 2024-04-23T18:27:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa23019dd30 con 0x7fa230106150 2024-04-23T18:27:49.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224035070 con 0x7fa230106150 2024-04-23T18:27:49.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.746+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2301a54f0 con 0x7fa230106150 2024-04-23T18:27:49.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.747+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa230068d70 con 0x7fa230106150 2024-04-23T18:27:49.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.748+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa224042030 con 0x7fa230106150 2024-04-23T18:27:49.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.749+0000 7fa217fff640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 0x7fa2180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:49.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.749+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fa2240f1030 con 0x7fa230106150 2024-04-23T18:27:49.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.749+0000 7fa22effd640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 0x7fa2180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:49.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.750+0000 7fa22effd640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 0x7fa2180a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bbe0 tx=0x7fa22000b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:49.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.752+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2240bd6d0 con 0x7fa230106150 2024-04-23T18:27:49.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.937+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa230101a30 con 0x7fa230106150 2024-04-23T18:27:49.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.938+0000 7fa217fff640 1 -- 172.21.15.26:0/4282288790 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7fa2240bcfd0 con 0x7fa230106150 2024-04-23T18:27:49.931 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 32 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 msgr2=0x7fa2180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 0x7fa2180a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bbe0 tx=0x7fa22000b390 comp rx=0 tx=0).stop 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 msgr2=0x7fa230198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa224006010 tx=0x7fa224004060 comp rx=0 tx=0).stop 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 shutdown_connections 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fa2180a3d20 0x7fa2180a61e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa23019cfb0 0x7fa23019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fa230106150 0x7fa230198c30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 --2- 172.21.15.26:0/4282288790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa230101080 0x7fa2301986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:49.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 >> 172.21.15.26:0/4282288790 conn(0x7fa230078000 msgr2=0x7fa2300fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:49.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 shutdown_connections 2024-04-23T18:27:49.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:49.940+0000 7fa2353bf640 1 -- 172.21.15.26:0/4282288790 wait complete. 2024-04-23T18:27:49.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:49 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/710716521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:49.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:49 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1268410200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:49.942 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:49 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/710716521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:49.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:49 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1268410200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:49.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:27:49.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:49 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/710716521' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:49.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:49 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1268410200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- 172.21.15.26:0/71450675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608101090 msgr2=0x7fe608101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 --2- 172.21.15.26:0/71450675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608101090 0x7fe608101470 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe604009900 tx=0x7fe60402a410 comp rx=0 tx=0).stop 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- 172.21.15.26:0/71450675 shutdown_connections 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 --2- 172.21.15.26:0/71450675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101a40 0x7fe608105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 --2- 172.21.15.26:0/71450675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe608106160 0x7fe60810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 --2- 172.21.15.26:0/71450675 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608101090 0x7fe608101470 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- 172.21.15.26:0/71450675 >> 172.21.15.26:0/71450675 conn(0x7fe608077f50 msgr2=0x7fe608078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- 172.21.15.26:0/71450675 shutdown_connections 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- 172.21.15.26:0/71450675 wait complete. 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 Processor -- start 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 -- start start 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.056+0000 7fe610764640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 0x7fe60810e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe60810f140 0x7fe6081a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fe6081121e0 con 0x7fe608106160 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe608112060 con 0x7fe608101090 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fe608112360 con 0x7fe60810f140 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33004/0 (socket says 172.21.15.26:33004) 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 -- 172.21.15.26:0/3229771835 learned_addr learned my addr 172.21.15.26:0/3229771835 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60e500640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 0x7fe60810e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 -- 172.21.15.26:0/3229771835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe60810f140 msgr2=0x7fe6081a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60ed01640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe60810f140 0x7fe6081a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe60810f140 0x7fe6081a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 -- 172.21.15.26:0/3229771835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 msgr2=0x7fe60810e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 0x7fe60810e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 -- 172.21.15.26:0/3229771835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6081a6f70 con 0x7fe608106160 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60e500640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 0x7fe60810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe60dcff640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800c970 tx=0x7fe5f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8014070 con 0x7fe608106160 2024-04-23T18:27:50.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6081a7260 con 0x7fe608106160 2024-04-23T18:27:50.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5f8004440 con 0x7fe608106160 2024-04-23T18:27:50.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8003e50 con 0x7fe608106160 2024-04-23T18:27:50.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.057+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6081a7800 con 0x7fe608106160 2024-04-23T18:27:50.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.058+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe608068d70 con 0x7fe608106160 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.059+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe5f8017070 con 0x7fe608106160 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.060+0000 7fe5ff7fe640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 0x7fe5f40a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.060+0000 7fe60e500640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 0x7fe5f40a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.060+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fe5f80d0be0 con 0x7fe608106160 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.061+0000 7fe60e500640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 0x7fe5f40a63b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe604000c00 tx=0x7fe604031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:50.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.064+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5f801e030 con 0x7fe608106160 2024-04-23T18:27:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.252+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe608109390 con 0x7fe608106160 2024-04-23T18:27:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.253+0000 7fe5ff7fe640 1 -- 172.21.15.26:0/3229771835 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7fe5f809d220 con 0x7fe608106160 2024-04-23T18:27:50.246 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 64 2024-04-23T18:27:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 msgr2=0x7fe5f40a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 0x7fe5f40a63b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe604000c00 tx=0x7fe604031400 comp rx=0 tx=0).stop 2024-04-23T18:27:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 msgr2=0x7fe60810ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800c970 tx=0x7fe5f800ce30 comp rx=0 tx=0).stop 2024-04-23T18:27:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 shutdown_connections 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fe5f40a3ef0 0x7fe5f40a63b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe60810f140 0x7fe6081a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fe608106160 0x7fe60810ec00 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 --2- 172.21.15.26:0/3229771835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe608101090 0x7fe60810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 >> 172.21.15.26:0/3229771835 conn(0x7fe608077f50 msgr2=0x7fe6080fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 shutdown_connections 2024-04-23T18:27:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.255+0000 7fe610764640 1 -- 172.21.15.26:0/3229771835 wait complete. 2024-04-23T18:27:50.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 -- 172.21.15.26:0/2983796659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610107240 msgr2=0x7f361010ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 --2- 172.21.15.26:0/2983796659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610107240 0x7f361010ee10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3608009ca0 tx=0x7f360802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 -- 172.21.15.26:0/2983796659 shutdown_connections 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 --2- 172.21.15.26:0/2983796659 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610107240 0x7f361010ee10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 --2- 172.21.15.26:0/2983796659 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610102b20 0x7f3610106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 --2- 172.21.15.26:0/2983796659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3610102170 0x7f3610102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 -- 172.21.15.26:0/2983796659 >> 172.21.15.26:0/2983796659 conn(0x7f3610079310 msgr2=0x7f3610079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 -- 172.21.15.26:0/2983796659 shutdown_connections 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.370+0000 7f3618136640 1 -- 172.21.15.26:0/2983796659 wait complete. 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 Processor -- start 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 -- start start 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610107240 0x7f3610198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 0x7f361019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f36101121b0 con 0x7f3610102170 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3610112030 con 0x7f361019cfb0 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3618136640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f3610112330 con 0x7f3610107240 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33030/0 (socket says 172.21.15.26:33030) 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 -- 172.21.15.26:0/761340423 learned_addr learned my addr 172.21.15.26:0/761340423 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f36156d1640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610107240 0x7f3610198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 -- 172.21.15.26:0/761340423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610107240 msgr2=0x7f3610198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610107240 0x7f3610198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 -- 172.21.15.26:0/761340423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 msgr2=0x7f361019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f36166d3640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 0x7f361019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 0x7f361019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 -- 172.21.15.26:0/761340423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361019daa0 con 0x7f3610102170 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f36166d3640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 0x7f361019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.371+0000 7f3615ed2640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f360c00e9f0 tx=0x7f360c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.372+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c00cd80 con 0x7f3610102170 2024-04-23T18:27:50.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.372+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361019dd90 con 0x7f3610102170 2024-04-23T18:27:50.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.372+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360c0108e0 con 0x7f3610102170 2024-04-23T18:27:50.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.372+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c018980 con 0x7f3610102170 2024-04-23T18:27:50.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.372+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36101a5580 con 0x7f3610102170 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.373+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36100691c0 con 0x7f3610102170 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.374+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f360c010430 con 0x7f3610102170 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.374+0000 7f3606ffd640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 0x7f35fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.375+0000 7f36156d1640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 0x7f35fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.375+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f360c0d11a0 con 0x7f3610102170 2024-04-23T18:27:50.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.375+0000 7f36156d1640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 0x7f35fc0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3610101ab0 tx=0x7f3600005f90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:50.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.378+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f360c09d7e0 con 0x7f3610102170 2024-04-23T18:27:50.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.576+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3610199a40 con 0x7f3610102170 2024-04-23T18:27:50.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.577+0000 7f3606ffd640 1 -- 172.21.15.26:0/761340423 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v98) ==== 81+0+940 (secure 0 0 0) 0x7f360c09d0e0 con 0x7f3610102170 2024-04-23T18:27:50.570 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/270207440 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.570 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/1107719668 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.570 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/270207440 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.570 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2404051623 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.570 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3591553425 2024-04-24T18:08:10.757945+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2032390504 2024-04-24T18:08:10.757945+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/497279081 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/2629443403 2024-04-24T18:08:10.757945+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/736937134 2024-04-24T18:17:09.435873+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/2629443403 2024-04-24T18:08:10.757945+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/4146543668 2024-04-24T18:07:44.350411+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/3418832937 2024-04-24T18:07:44.350411+0000 2024-04-23T18:27:50.571 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/2243122498 2024-04-24T18:07:44.350411+0000 2024-04-23T18:27:50.572 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6800/259946325 2024-04-24T18:07:44.350411+0000 2024-04-23T18:27:50.572 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:6801/259946325 2024-04-24T18:07:44.350411+0000 2024-04-23T18:27:50.572 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26:0/1804299815 2024-04-24T18:08:10.757945+0000 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:listed 16 entries 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 msgr2=0x7f35fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 0x7f35fc0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3610101ab0 tx=0x7f3600005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 msgr2=0x7f36101986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f360c00e9f0 tx=0x7f360c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 shutdown_connections 2024-04-23T18:27:50.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f35fc0a3d20 0x7f35fc0a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361019cfb0 0x7f361019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3610107240 0x7f3610198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 --2- 172.21.15.26:0/761340423 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f3610102170 0x7f36101986f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 >> 172.21.15.26:0/761340423 conn(0x7f3610079310 msgr2=0x7f3610100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 shutdown_connections 2024-04-23T18:27:50.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:50.579+0000 7f3618136640 1 -- 172.21.15.26:0/761340423 wait complete. 2024-04-23T18:27:50.583 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T18:27:50.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:27:50.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-23T18:27:50.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:27:50.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:27:50.651 DEBUG:teuthology.orchestra.run.smithi026:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-23T18:27:50.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:27:50.687 DEBUG:teuthology.orchestra.run.smithi026:> ip netns list 2024-04-23T18:27:50.759 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-23T18:27:50.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:27:50.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T18:27:50.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:27:50.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo ip link delete ceph-brx 2024-04-23T18:27:50.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:50 smithi026 ceph-mon[29364]: pgmap v332: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:50.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:50 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4282288790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.834 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:50 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3229771835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.834 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:50 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/761340423' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:27:50.883 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T18:27:50.893 INFO:tasks.cephadm:Teardown begin 2024-04-23T18:27:50.894 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T18:27:50.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:50.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:50 smithi110 ceph-mon[32585]: pgmap v332: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:50.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:50 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4282288790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:50 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3229771835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:50 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/761340423' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:27:50.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:50 smithi131 ceph-mon[33041]: pgmap v332: 97 pgs: 97 active+clean; 357 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2024-04-23T18:27:50.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:50 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4282288790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:50 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3229771835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:27:50.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:50 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/761340423' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:27:51.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 -- 172.21.15.26:0/2259010195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d4102bf0 msgr2=0x7f79d4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 --2- 172.21.15.26:0/2259010195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d4102bf0 0x7f79d4106ba0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009900 tx=0x7f79c802a410 comp rx=0 tx=0).stop 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 -- 172.21.15.26:0/2259010195 shutdown_connections 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 --2- 172.21.15.26:0/2259010195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 0x7f79d410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 --2- 172.21.15.26:0/2259010195 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d4102bf0 0x7f79d4106ba0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 --2- 172.21.15.26:0/2259010195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d4102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 -- 172.21.15.26:0/2259010195 >> 172.21.15.26:0/2259010195 conn(0x7f79d40790e0 msgr2=0x7f79d40794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 -- 172.21.15.26:0/2259010195 shutdown_connections 2024-04-23T18:27:51.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.005+0000 7f79da010640 1 -- 172.21.15.26:0/2259010195 wait complete. 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 Processor -- start 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 -- start start 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 0x7f79d419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4112210 con 0x7f79d41985d0 2024-04-23T18:27:51.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4112090 con 0x7f79d4102240 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79da010640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4112390 con 0x7f79d4107280 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.006+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33054/0 (socket says 172.21.15.26:33054) 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.26:37948/0 (socket says 172.21.15.26:37948) 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 -- 172.21.15.26:0/1934367397 learned_addr learned my addr 172.21.15.26:0/1934367397 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d2ffd640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 0x7f79d419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 -- 172.21.15.26:0/1934367397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 msgr2=0x7f79d419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 0x7f79d419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 -- 172.21.15.26:0/1934367397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 msgr2=0x7f79d419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 -- 172.21.15.26:0/1934367397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d41990f0 con 0x7f79d41985d0 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d3fff640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cad0 tx=0x7f79cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00ae20 con 0x7f79d41985d0 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79cc005020 con 0x7f79d41985d0 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d41993e0 con 0x7f79d41985d0 2024-04-23T18:27:51.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00f6b0 con 0x7f79d41985d0 2024-04-23T18:27:51.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.007+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d41a5680 con 0x7f79d41985d0 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.008+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d40691c0 con 0x7f79d41985d0 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.009+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f79cc0051d0 con 0x7f79d41985d0 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.010+0000 7f79d0ff9640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 0x7f79bc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.010+0000 7f79d37fe640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 0x7f79bc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.010+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f79cc0d0730 con 0x7f79d41985d0 2024-04-23T18:27:51.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.010+0000 7f79d37fe640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 0x7f79bc0a6260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f79c40058c0 tx=0x7f79c400a2d0 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.014+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79cc014030 con 0x7f79d41985d0 2024-04-23T18:27:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.249+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79d4102bf0 con 0x7f79d41985d0 2024-04-23T18:27:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.249+0000 7f79d0ff9640 1 -- 172.21.15.26:0/1934367397 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11890 (secure 0 0 0) 0x7f79cc09cdd0 con 0x7f79d41985d0 2024-04-23T18:27:51.242 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:51.242 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:27:02.123441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:27:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 49 2024-04-23T18:27:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 msgr2=0x7f79bc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 0x7f79bc0a6260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f79c40058c0 tx=0x7f79c400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 msgr2=0x7f79d4198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cad0 tx=0x7f79cc00cf90 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 shutdown_connections 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f79bc0a3da0 0x7f79bc0a6260 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f79d41985d0 0x7f79d4198a80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79d4107280 0x7f79d419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 --2- 172.21.15.26:0/1934367397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79d4102240 0x7f79d419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 >> 172.21.15.26:0/1934367397 conn(0x7f79d40790e0 msgr2=0x7f79d4100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 shutdown_connections 2024-04-23T18:27:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.252+0000 7f79da010640 1 -- 172.21.15.26:0/1934367397 wait complete. 2024-04-23T18:27:51.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 -- 172.21.15.26:0/2323796787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 msgr2=0x7fd2e810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 --2- 172.21.15.26:0/2323796787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e810ee30 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009ca0 tx=0x7fd2e002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 -- 172.21.15.26:0/2323796787 shutdown_connections 2024-04-23T18:27:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 --2- 172.21.15.26:0/2323796787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e8102b40 0x7fd2e8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 --2- 172.21.15.26:0/2323796787 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e810ee30 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 --2- 172.21.15.26:0/2323796787 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 0x7fd2e8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 -- 172.21.15.26:0/2323796787 >> 172.21.15.26:0/2323796787 conn(0x7fd2e8079310 msgr2=0x7fd2e8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 -- 172.21.15.26:0/2323796787 shutdown_connections 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.366+0000 7fd2eda42640 1 -- 172.21.15.26:0/2323796787 wait complete. 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 Processor -- start 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 -- start start 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 0x7fd2e8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 0x7fd2e819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e81121d0 con 0x7fd2e8107260 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8112050 con 0x7fd2e819cfe0 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2eda42640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8112350 con 0x7fd2e8102190 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33076/0 (socket says 172.21.15.26:33076) 2024-04-23T18:27:51.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 -- 172.21.15.26:0/407147129 learned_addr learned my addr 172.21.15.26:0/407147129 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e77fe640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 0x7fd2e819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e6ffd640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 0x7fd2e8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 -- 172.21.15.26:0/407147129 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 msgr2=0x7fd2e8198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 0x7fd2e8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 -- 172.21.15.26:0/407147129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 msgr2=0x7fd2e819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.367+0000 7fd2e67fc640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 0x7fd2e819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2e67fc640 1 -- 172.21.15.26:0/407147129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e819dad0 con 0x7fd2e8107260 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2e77fe640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 0x7fd2e819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:51.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2e67fc640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc005fd0 tx=0x7fd2dc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc017070 con 0x7fd2e8107260 2024-04-23T18:27:51.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2dc009c30 con 0x7fd2e8107260 2024-04-23T18:27:51.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e819ddc0 con 0x7fd2e8107260 2024-04-23T18:27:51.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc003d30 con 0x7fd2e8107260 2024-04-23T18:27:51.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.368+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e81a55b0 con 0x7fd2e8107260 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.369+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e80691c0 con 0x7fd2e8107260 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.370+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd2dc003ed0 con 0x7fd2e8107260 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.371+0000 7fd2eca40640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 0x7fd2d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.371+0000 7fd2e6ffd640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 0x7fd2d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.371+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fd2dc0d04b0 con 0x7fd2e8107260 2024-04-23T18:27:51.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.371+0000 7fd2e6ffd640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 0x7fd2d00a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005fd0 tx=0x7fd2d8005850 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.376+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2dc020030 con 0x7fd2e8107260 2024-04-23T18:27:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.603+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e8102b40 con 0x7fd2e8107260 2024-04-23T18:27:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.604+0000 7fd2eca40640 1 -- 172.21.15.26:0/407147129 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11890 (secure 0 0 0) 0x7fd2dc09cb50 con 0x7fd2e8107260 2024-04-23T18:27:51.597 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:51.597 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6835/1467060531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6834","nonce":1467060531},{"type":"v1","addr":"172.21.15.26:6835","nonce":1467060531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6837/1500031392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6836","nonce":1500031392},{"type":"v1","addr":"172.21.15.26:6837","nonce":1500031392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6839/1936694091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6838","nonce":1936694091},{"type":"v1","addr":"172.21.15.26:6839","nonce":1936694091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.26:6841/2731986985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6840","nonce":2731986985},{"type":"v1","addr":"172.21.15.26:6841","nonce":2731986985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/2977568812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":2977568812},{"type":"v1","addr":"172.21.15.131:6833","nonce":2977568812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/1312481724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1312481724},{"type":"v1","addr":"172.21.15.110:6833","nonce":1312481724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:15:06.048360+0000","modified":"2024-04-23T18:27:02.123441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24464,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/2220145154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":2220145154},{"type":"v1","addr":"172.21.15.131:6835","nonce":2220145154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/3666168856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":3666168856},{"type":"v1","addr":"172.21.15.110:6835","nonce":3666168856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.131:6839/4056718895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":4056718895},{"type":"v1","addr":"172.21.15.131:6839","nonce":4056718895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6837/3430274697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":3430274697},{"type":"v1","addr":"172.21.15.110:6837","nonce":3430274697}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/3619972803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":3619972803},{"type":"v1","addr":"172.21.15.110:6839","nonce":3619972803}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25237":{"gid":25237,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6837/2307422363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2307422363},{"type":"v1","addr":"172.21.15.131:6837","nonce":2307422363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24460,24464,24503]},"id":1}]} 2024-04-23T18:27:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:dumped fsmap epoch 49 2024-04-23T18:27:51.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 msgr2=0x7fd2d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 0x7fd2d00a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005fd0 tx=0x7fd2d8005850 comp rx=0 tx=0).stop 2024-04-23T18:27:51.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 msgr2=0x7fd2e8198cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc005fd0 tx=0x7fd2dc00dde0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 shutdown_connections 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fd2d00a3d20 0x7fd2d00a61e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2e819cfe0 0x7fd2e819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fd2e8107260 0x7fd2e8198cb0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 --2- 172.21.15.26:0/407147129 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2e8102190 0x7fd2e8198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 >> 172.21.15.26:0/407147129 conn(0x7fd2e8079310 msgr2=0x7fd2e8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 shutdown_connections 2024-04-23T18:27:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.606+0000 7fd2eda42640 1 -- 172.21.15.26:0/407147129 wait complete. 2024-04-23T18:27:51.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:27:51.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 -- 172.21.15.26:0/1525318023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4101ac0 msgr2=0x7f80b4105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 --2- 172.21.15.26:0/1525318023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4101ac0 0x7f80b4105ab0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f809c009900 tx=0x7f809c02a410 comp rx=0 tx=0).stop 2024-04-23T18:27:51.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 -- 172.21.15.26:0/1525318023 shutdown_connections 2024-04-23T18:27:51.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 --2- 172.21.15.26:0/1525318023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4106190 0x7f80b410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 --2- 172.21.15.26:0/1525318023 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4101ac0 0x7f80b4105ab0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 --2- 172.21.15.26:0/1525318023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b4101110 0x7f80b41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 -- 172.21.15.26:0/1525318023 >> 172.21.15.26:0/1525318023 conn(0x7f80b4077fb0 msgr2=0x7f80b40783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 -- 172.21.15.26:0/1525318023 shutdown_connections 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.720+0000 7f80b978e640 1 -- 172.21.15.26:0/1525318023 wait complete. 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 Processor -- start 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 -- start start 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4101110 0x7f80b4102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 0x7f80b41a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4111100 con 0x7f80b4106190 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4110f80 con 0x7f80b4101110 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b978e640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4111280 con 0x7f80b41035d0 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33104/0 (socket says 172.21.15.26:33104) 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 -- 172.21.15.26:0/3667033324 learned_addr learned my addr 172.21.15.26:0/3667033324 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b2ffd640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4101110 0x7f80b4102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 -- 172.21.15.26:0/3667033324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 msgr2=0x7f80b41a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b37fe640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 0x7f80b41a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 0x7f80b41a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 -- 172.21.15.26:0/3667033324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4101110 msgr2=0x7f80b4102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4101110 0x7f80b4102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80aa7fc640 1 -- 172.21.15.26:0/3667033324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b41a6fe0 con 0x7f80b4106190 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.721+0000 7f80b37fe640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 0x7f80b41a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80aa7fc640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f809c005ab0 tx=0x7f809c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c03f070 con 0x7f80b4106190 2024-04-23T18:27:51.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f809c031d60 con 0x7f80b4106190 2024-04-23T18:27:51.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b41a7270 con 0x7f80b4106190 2024-04-23T18:27:51.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c035030 con 0x7f80b4106190 2024-04-23T18:27:51.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.722+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b41a77b0 con 0x7f80b4106190 2024-04-23T18:27:51.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.723+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b40691c0 con 0x7f80b4106190 2024-04-23T18:27:51.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.724+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f809c042070 con 0x7f80b4106190 2024-04-23T18:27:51.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.725+0000 7f80b0ff9640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 0x7f80a00a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:51.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.725+0000 7f80b2ffd640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 0x7f80a00a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.725+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f809c0f0f10 con 0x7f80b4106190 2024-04-23T18:27:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.725+0000 7f80b2ffd640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 0x7f80a00a6290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f80a400bbe0 tx=0x7f80a400b390 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.730+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f809c0bd5b0 con 0x7f80b4106190 2024-04-23T18:27:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.919+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80b410a3d0 con 0x7f80b4106190 2024-04-23T18:27:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.920+0000 7f80b0ff9640 1 -- 172.21.15.26:0/3667033324 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+23127 (secure 0 0 0) 0x7f809c0bceb0 con 0x7f80b4106190 2024-04-23T18:27:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-23T18:27:51.913 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":98,"fsid":"2a701120-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:07:16.481986+0000","modified":"2024-04-23T18:27:16.798801+0000","last_up_change":"2024-04-23T18:14:15.726016+0000","last_in_change":"2024-04-23T18:13:55.091054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:10:34.851766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:15:03.735783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:15:04.991252+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6232862c-582d-4b53-96c1-a331e0e5849d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6803","nonce":75846197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6805","nonce":75846197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6809","nonce":75846197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":75846197},{"type":"v1","addr":"172.21.15.26:6807","nonce":75846197}]},"public_addr":"172.21.15.26:6803/75846197","cluster_addr":"172.21.15.26:6805/75846197","heartbeat_back_addr":"172.21.15.26:6809/75846197","heartbeat_front_addr":"172.21.15.26:6807/75846197","state":["exists","up"]},{"osd":1,"uuid":"ea35a550-d935-4416-9e66-29ae820b9e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6801","nonce":1354854884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6803","nonce":1354854884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6807","nonce":1354854884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1354854884},{"type":"v1","addr":"172.21.15.110:6805","nonce":1354854884}]},"public_addr":"172.21.15.110:6801/1354854884","cluster_addr":"172.21.15.110:6803/1354854884","heartbeat_back_addr":"172.21.15.110:6807/1354854884","heartbeat_front_addr":"172.21.15.110:6805/1354854884","state":["exists","up"]},{"osd":2,"uuid":"09b22982-126d-46bf-9e92-d1792b8c8477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6801","nonce":2462004280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6803","nonce":2462004280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6807","nonce":2462004280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2462004280},{"type":"v1","addr":"172.21.15.131:6805","nonce":2462004280}]},"public_addr":"172.21.15.131:6801/2462004280","cluster_addr":"172.21.15.131:6803/2462004280","heartbeat_back_addr":"172.21.15.131:6807/2462004280","heartbeat_front_addr":"172.21.15.131:6805/2462004280","state":["exists","up"]},{"osd":3,"uuid":"3f222d1e-da51-4d64-8e76-53e764356085","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6811","nonce":3619192786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6813","nonce":3619192786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6817","nonce":3619192786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3619192786},{"type":"v1","addr":"172.21.15.26:6815","nonce":3619192786}]},"public_addr":"172.21.15.26:6811/3619192786","cluster_addr":"172.21.15.26:6813/3619192786","heartbeat_back_addr":"172.21.15.26:6817/3619192786","heartbeat_front_addr":"172.21.15.26:6815/3619192786","state":["exists","up"]},{"osd":4,"uuid":"3834c24f-b7b4-41f8-8878-1c54b691ff48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6809","nonce":2989079623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6811","nonce":2989079623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6815","nonce":2989079623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2989079623},{"type":"v1","addr":"172.21.15.110:6813","nonce":2989079623}]},"public_addr":"172.21.15.110:6809/2989079623","cluster_addr":"172.21.15.110:6811/2989079623","heartbeat_back_addr":"172.21.15.110:6815/2989079623","heartbeat_front_addr":"172.21.15.110:6813/2989079623","state":["exists","up"]},{"osd":5,"uuid":"12b632a3-1a0b-4918-9a4d-ccc144053d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6809","nonce":3617441871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6811","nonce":3617441871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6815","nonce":3617441871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":3617441871},{"type":"v1","addr":"172.21.15.131:6813","nonce":3617441871}]},"public_addr":"172.21.15.131:6809/3617441871","cluster_addr":"172.21.15.131:6811/3617441871","heartbeat_back_addr":"172.21.15.131:6815/3617441871","heartbeat_front_addr":"172.21.15.131:6813/3617441871","state":["exists","up"]},{"osd":6,"uuid":"6f5e9d18-f986-4bd6-87b6-ceb84b684bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6819","nonce":3473982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6821","nonce":3473982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6825","nonce":3473982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3473982661},{"type":"v1","addr":"172.21.15.26:6823","nonce":3473982661}]},"public_addr":"172.21.15.26:6819/3473982661","cluster_addr":"172.21.15.26:6821/3473982661","heartbeat_back_addr":"172.21.15.26:6825/3473982661","heartbeat_front_addr":"172.21.15.26:6823/3473982661","state":["exists","up"]},{"osd":7,"uuid":"f090001e-2004-4cf8-bcf2-38a16b9618e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6817","nonce":3458351312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6819","nonce":3458351312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6823","nonce":3458351312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3458351312},{"type":"v1","addr":"172.21.15.110:6821","nonce":3458351312}]},"public_addr":"172.21.15.110:6817/3458351312","cluster_addr":"172.21.15.110:6819/3458351312","heartbeat_back_addr":"172.21.15.110:6823/3458351312","heartbeat_front_addr":"172.21.15.110:6821/3458351312","state":["exists","up"]},{"osd":8,"uuid":"c05d9ea6-d3bb-4333-a68a-f2d689e41310","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6817","nonce":3490838385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6819","nonce":3490838385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6823","nonce":3490838385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":3490838385},{"type":"v1","addr":"172.21.15.131:6821","nonce":3490838385}]},"public_addr":"172.21.15.131:6817/3490838385","cluster_addr":"172.21.15.131:6819/3490838385","heartbeat_back_addr":"172.21.15.131:6823/3490838385","heartbeat_front_addr":"172.21.15.131:6821/3490838385","state":["exists","up"]},{"osd":9,"uuid":"34848530-043d-4447-a445-4c1e4d5f1c8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6827","nonce":1397303359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6829","nonce":1397303359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6833","nonce":1397303359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":1397303359},{"type":"v1","addr":"172.21.15.26:6831","nonce":1397303359}]},"public_addr":"172.21.15.26:6827/1397303359","cluster_addr":"172.21.15.26:6829/1397303359","heartbeat_back_addr":"172.21.15.26:6833/1397303359","heartbeat_front_addr":"172.21.15.26:6831/1397303359","state":["exists","up"]},{"osd":10,"uuid":"d5bcbd75-0ca4-450e-b735-3bcb342f53a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6825","nonce":3721710619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6827","nonce":3721710619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6831","nonce":3721710619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":3721710619},{"type":"v1","addr":"172.21.15.110:6829","nonce":3721710619}]},"public_addr":"172.21.15.110:6825/3721710619","cluster_addr":"172.21.15.110:6827/3721710619","heartbeat_back_addr":"172.21.15.110:6831/3721710619","heartbeat_front_addr":"172.21.15.110:6829/3721710619","state":["exists","up"]},{"osd":11,"uuid":"e17072b7-66af-42ed-a40e-7b472fa918f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6825","nonce":2935357051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6827","nonce":2935357051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6831","nonce":2935357051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2935357051},{"type":"v1","addr":"172.21.15.131:6829","nonce":2935357051}]},"public_addr":"172.21.15.131:6825/2935357051","cluster_addr":"172.21.15.131:6827/2935357051","heartbeat_back_addr":"172.21.15.131:6831/2935357051","heartbeat_front_addr":"172.21.15.131:6829/2935357051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:09:44.819004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:08.600906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:32.028379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:10:56.871156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:21.101213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:44.904882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:10.152704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:34.642611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:58.006040+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:23.581350+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:48.250244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:13.022146+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/1107719668":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/270207440":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/2404051623":"2024-04-24T18:17:09.435873+0000","172.21.15.26:0/3591553425":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/2032390504":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/497279081":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6801/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/736937134":"2024-04-24T18:17:09.435873+0000","172.21.15.26:6800/2629443403":"2024-04-24T18:08:10.757945+0000","172.21.15.26:0/4146543668":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/3418832937":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/2243122498":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6800/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:6801/259946325":"2024-04-24T18:07:44.350411+0000","172.21.15.26:0/1804299815":"2024-04-24T18:08:10.757945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:27:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 msgr2=0x7f80a00a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 0x7f80a00a6290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f80a400bbe0 tx=0x7f80a400b390 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 msgr2=0x7f80b4103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f809c005ab0 tx=0x7f809c003cd0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 shutdown_connections 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f80a00a3dd0 0x7f80a00a6290 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80b41035d0 0x7f80b41a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f80b4106190 0x7f80b4103090 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 --2- 172.21.15.26:0/3667033324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80b4101110 0x7f80b4102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 >> 172.21.15.26:0/3667033324 conn(0x7f80b4077fb0 msgr2=0x7f80b406a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 shutdown_connections 2024-04-23T18:27:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:51.923+0000 7f80b978e640 1 -- 172.21.15.26:0/3667033324 wait complete. 2024-04-23T18:27:51.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:51 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/1934367397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:51.926 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:51 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/407147129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:51.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T18:27:51.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:51 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/1934367397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:51.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:51 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/407147129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:51.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:51 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/1934367397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:51.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:51 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/407147129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:52.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 -- 172.21.15.26:0/398960209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c103b50 msgr2=0x7f0d2c10c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 --2- 172.21.15.26:0/398960209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c103b50 0x7f0d2c10c030 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c009900 tx=0x7f0d1c02a480 comp rx=0 tx=0).stop 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 -- 172.21.15.26:0/398960209 shutdown_connections 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 --2- 172.21.15.26:0/398960209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c10c600 0x7f0d2c10ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 --2- 172.21.15.26:0/398960209 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c103b50 0x7f0d2c10c030 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 --2- 172.21.15.26:0/398960209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103230 0x7f0d2c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 -- 172.21.15.26:0/398960209 >> 172.21.15.26:0/398960209 conn(0x7f0d2c077f80 msgr2=0x7f0d2c078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 -- 172.21.15.26:0/398960209 shutdown_connections 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.037+0000 7f0d328cf640 1 -- 172.21.15.26:0/398960209 wait complete. 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 Processor -- start 2024-04-23T18:27:52.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 -- start start 2024-04-23T18:27:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c103230 0x7f0d2c100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103b50 0x7f0d2c1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c112220 con 0x7f0d2c10c600 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c1120a0 con 0x7f0d2c103230 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d328cf640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c1123a0 con 0x7f0d2c103b50 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d30e6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d30e6c640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33112/0 (socket says 172.21.15.26:33112) 2024-04-23T18:27:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d30e6c640 1 -- 172.21.15.26:0/3192054455 learned_addr learned my addr 172.21.15.26:0/3192054455 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d2b7fe640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103b50 0x7f0d2c1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.038+0000 7f0d2bfff640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c103230 0x7f0d2c100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 -- 172.21.15.26:0/3192054455 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103b50 msgr2=0x7f0d2c1013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103b50 0x7f0d2c1013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 -- 172.21.15.26:0/3192054455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c103230 msgr2=0x7f0d2c100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c103230 0x7f0d2c100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 -- 172.21.15.26:0/3192054455 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c1a6ca0 con 0x7f0d2c10c600 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d30e6c640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400eaa0 tx=0x7f0d2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400cdb0 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d24005020 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2c1a6f90 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240116d0 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.039+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2c1a7460 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.040+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2c06add0 con 0x7f0d2c10c600 2024-04-23T18:27:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.041+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0d240051d0 con 0x7f0d2c10c600 2024-04-23T18:27:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.042+0000 7f0d297fa640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 0x7f0d140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.042+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f0d240d0cd0 con 0x7f0d2c10c600 2024-04-23T18:27:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.042+0000 7f0d2bfff640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 0x7f0d140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.042+0000 7f0d2bfff640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 0x7f0d140a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005fd0 tx=0x7f0d2000b590 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.045+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d24016030 con 0x7f0d2c10c600 2024-04-23T18:27:52.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.270+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0d2c0691c0 con 0x7f0d2c10c600 2024-04-23T18:27:52.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.273+0000 7f0d297fa640 1 -- 172.21.15.26:0/3192054455 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 161+0+0 (secure 0 0 0) 0x7f0d2c0691c0 con 0x7f0d2c10c600 2024-04-23T18:27:52.265 INFO:teuthology.orchestra.run.smithi026.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T18:27:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.275+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 msgr2=0x7f0d140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.275+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 0x7f0d140a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005fd0 tx=0x7f0d2000b590 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.275+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 msgr2=0x7f0d2c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.275+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400eaa0 tx=0x7f0d2400ef60 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 shutdown_connections 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f0d140a3d20 0x7f0d140a61e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f0d2c10c600 0x7f0d2c101900 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0d2c103b50 0x7f0d2c1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 --2- 172.21.15.26:0/3192054455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d2c103230 0x7f0d2c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 >> 172.21.15.26:0/3192054455 conn(0x7f0d2c077f80 msgr2=0x7f0d2c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 shutdown_connections 2024-04-23T18:27:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.276+0000 7f0d328cf640 1 -- 172.21.15.26:0/3192054455 wait complete. 2024-04-23T18:27:52.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T18:27:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- 172.21.15.26:0/1982865585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a64101090 msgr2=0x7f2a64101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 --2- 172.21.15.26:0/1982865585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a64101090 0x7f2a64101470 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009ca0 tx=0x7f2a5402c790 comp rx=0 tx=0).stop 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- 172.21.15.26:0/1982865585 shutdown_connections 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 --2- 172.21.15.26:0/1982865585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64106160 0x7f2a6410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 --2- 172.21.15.26:0/1982865585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64101a40 0x7f2a64105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 --2- 172.21.15.26:0/1982865585 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a64101090 0x7f2a64101470 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- 172.21.15.26:0/1982865585 >> 172.21.15.26:0/1982865585 conn(0x7f2a64077f50 msgr2=0x7f2a64078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- 172.21.15.26:0/1982865585 shutdown_connections 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- 172.21.15.26:0/1982865585 wait complete. 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 Processor -- start 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.394+0000 7f2a69e79640 1 -- start start 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64101090 0x7f2a64196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 0x7f2a64196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f2a64110000 con 0x7f2a6419ae40 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6410fe80 con 0x7f2a64101090 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f2a64110180 con 0x7f2a64106160 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33122/0 (socket says 172.21.15.26:33122) 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 -- 172.21.15.26:0/3628680682 learned_addr learned my addr 172.21.15.26:0/3628680682 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a62ffd640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 0x7f2a64196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a637fe640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64101090 0x7f2a64196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 -- 172.21.15.26:0/3628680682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 msgr2=0x7f2a64196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 0x7f2a64196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 -- 172.21.15.26:0/3628680682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64101090 msgr2=0x7f2a64196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64101090 0x7f2a64196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 -- 172.21.15.26:0/3628680682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6419b930 con 0x7f2a6419ae40 2024-04-23T18:27:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a62ffd640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 0x7f2a64196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a63fff640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00e970 tx=0x7f2a5c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c00ccc0 con 0x7f2a6419ae40 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a5c00ce60 con 0x7f2a6419ae40 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c010430 con 0x7f2a6419ae40 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6419bc20 con 0x7f2a6419ae40 2024-04-23T18:27:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.395+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a641a33c0 con 0x7f2a6419ae40 2024-04-23T18:27:52.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.396+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a64068d70 con 0x7f2a6419ae40 2024-04-23T18:27:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.397+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a5c004ff0 con 0x7f2a6419ae40 2024-04-23T18:27:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.398+0000 7f2a60ff9640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 0x7f2a500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.398+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8727+0+0 (secure 0 0 0) 0x7f2a5c0d0d10 con 0x7f2a6419ae40 2024-04-23T18:27:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.398+0000 7f2a637fe640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 0x7f2a500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:52.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.399+0000 7f2a637fe640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 0x7f2a500a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a54006010 tx=0x7f2a5402cf90 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:52.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.402+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a5c09d2e0 con 0x7f2a6419ae40 2024-04-23T18:27:52.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:52.629+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2a64101a40 con 0x7f2a6419ae40 2024-04-23T18:27:52.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a[29360]: 2024-04-23T18:27:52.271+0000 7f5bde7ce700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:27:52.688 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.688 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.688 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mds[63040]: -- 172.21.15.110:0/456807785 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d78d25a40 con 0x556d78cf8c00 2024-04-23T18:27:52.688 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: starting mds.b at 2024-04-23T18:27:52.688 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.689 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.689 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mds[66888]: -- 172.21.15.131:0/3176945150 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2a5ee1a40 con 0x55d2a5eb4c00 2024-04-23T18:27:52.689 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: starting mds.c at 2024-04-23T18:27:52.689 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.689 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mds[62719]: -- 172.21.15.110:0/1233741985 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56193cc6fa40 con 0x56193cc43000 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: starting mds.h at 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mds[63757]: -- 172.21.15.131:0/3358371545 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560adb53fa40 con 0x560adb512c00 2024-04-23T18:27:52.690 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: starting mds.i at 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mds[62413]: -- 172.21.15.110:0/3887799498 <== mon.0 v2:172.21.15.26:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55689f65fa40 con 0x55689f632c00 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: starting mds.e at 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:52.691 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:52.692 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mds[63135]: -- 172.21.15.131:0/201078362 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560fa223ba40 con 0x560fa220e800 2024-04-23T18:27:52.692 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: starting mds.f at 2024-04-23T18:27:52.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: pgmap v333: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 11 op/s 2024-04-23T18:27:52.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3667033324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:52.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:27:52.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:27:52.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:27:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:27:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:27:52.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:52 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:27:52.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: pgmap v333: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 11 op/s 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3667033324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:27:52.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:52 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: pgmap v333: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 11 op/s 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3667033324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3192054455' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:52.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:27:52.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:52 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:27:53.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.274+0000 7f2a60ff9640 1 -- 172.21.15.26:0/3628680682 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) ==== 104+0+0 (secure 0 0 0) 0x7f2a5c09cbe0 con 0x7f2a6419ae40 2024-04-23T18:27:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 msgr2=0x7f2a500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 0x7f2a500a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a54006010 tx=0x7f2a5402cf90 comp rx=0 tx=0).stop 2024-04-23T18:27:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 msgr2=0x7f2a6419b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:53.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00e970 tx=0x7f2a5c00ed70 comp rx=0 tx=0).stop 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 shutdown_connections 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f2a500a3d20 0x7f2a500a61e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f2a6419ae40 0x7f2a6419b2f0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a64106160 0x7f2a64196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 --2- 172.21.15.26:0/3628680682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a64101090 0x7f2a64196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.278+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 >> 172.21.15.26:0/3628680682 conn(0x7f2a64077f50 msgr2=0x7f2a640ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.279+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 shutdown_connections 2024-04-23T18:27:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.279+0000 7f2a69e79640 1 -- 172.21.15.26:0/3628680682 wait complete. 2024-04-23T18:27:53.286 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T18:27:53.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- 172.21.15.26:0/3378613062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c106160 msgr2=0x7fb67c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 --2- 172.21.15.26:0/3378613062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c106160 0x7fb67c10dd30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009900 tx=0x7fb66c02a410 comp rx=0 tx=0).stop 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- 172.21.15.26:0/3378613062 shutdown_connections 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 --2- 172.21.15.26:0/3378613062 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c106160 0x7fb67c10dd30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 --2- 172.21.15.26:0/3378613062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101a40 0x7fb67c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 --2- 172.21.15.26:0/3378613062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c101090 0x7fb67c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- 172.21.15.26:0/3378613062 >> 172.21.15.26:0/3378613062 conn(0x7fb67c077f50 msgr2=0x7fb67c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:53.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- 172.21.15.26:0/3378613062 shutdown_connections 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- 172.21.15.26:0/3378613062 wait complete. 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 Processor -- start 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.407+0000 7fb683243640 1 -- start start 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101090 0x7fb67c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c106160 0x7fb67c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c111120 con 0x7fb67c19d000 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c110fa0 con 0x7fb67c106160 2024-04-23T18:27:53.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1112a0 con 0x7fb67c101090 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33144/0 (socket says 172.21.15.26:33144) 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 -- 172.21.15.26:0/2614763009 learned_addr learned my addr 172.21.15.26:0/2614763009 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb680fdf640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101090 0x7fb67c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb67bfff640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c106160 0x7fb67c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 -- 172.21.15.26:0/2614763009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101090 msgr2=0x7fb67c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101090 0x7fb67c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 -- 172.21.15.26:0/2614763009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c106160 msgr2=0x7fb67c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:53.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c106160 0x7fb67c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 -- 172.21.15.26:0/2614763009 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c19daf0 con 0x7fb67c19d000 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb6817e0640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb66c02a930 tx=0x7fb66c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c03f070 con 0x7fb67c19d000 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c19dd80 con 0x7fb67c19d000 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb66c003e20 con 0x7fb67c19d000 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.408+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c035030 con 0x7fb67c19d000 2024-04-23T18:27:53.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.409+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c1a5550 con 0x7fb67c19d000 2024-04-23T18:27:53.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.410+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c068d70 con 0x7fb67c19d000 2024-04-23T18:27:53.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.411+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb66c042070 con 0x7fb67c19d000 2024-04-23T18:27:53.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.411+0000 7fb679ffb640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 0x7fb6680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.412+0000 7fb680fdf640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 0x7fb6680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.412+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8727+0+0 (secure 0 0 0) 0x7fb66c0f1040 con 0x7fb67c19d000 2024-04-23T18:27:53.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.412+0000 7fb680fdf640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 0x7fb6680a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb674005fd0 tx=0x7fb674009500 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:53.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.416+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb66c0bd610 con 0x7fb67c19d000 2024-04-23T18:27:53.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:53.605+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb67c10a3a0 con 0x7fb67c19d000 2024-04-23T18:27:53.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: Cluster is now healthy 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6834/3472170941,v1:172.21.15.131:6835/3472170941] up:boot 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6836/2217260833,v1:172.21.15.110:6838/2217260833] up:boot 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6834/2253059255,v1:172.21.15.110:6837/2253059255] up:boot 2024-04-23T18:27:53.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.110:6835/3896928820,v1:172.21.15.110:6839/3896928820] up:boot 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6836/4234467521,v1:172.21.15.131:6839/4234467521] up:boot 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: mds.? [v2:172.21.15.131:6837/2717670530,v1:172.21.15.131:6838/2717670530] up:boot 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: fsmap 12 up:standby 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:53.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:53 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: Cluster is now healthy 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:27:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6834/3472170941,v1:172.21.15.131:6835/3472170941] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6836/2217260833,v1:172.21.15.110:6838/2217260833] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6834/2253059255,v1:172.21.15.110:6837/2253059255] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.110:6835/3896928820,v1:172.21.15.110:6839/3896928820] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6836/4234467521,v1:172.21.15.131:6839/4234467521] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: mds.? [v2:172.21.15.131:6837/2717670530,v1:172.21.15.131:6838/2717670530] up:boot 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: fsmap 12 up:standby 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:27:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:27:53.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:27:53.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:53.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:53 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:53.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:27:53.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: Cluster is now healthy 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/3628680682' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6834/3472170941,v1:172.21.15.131:6835/3472170941] up:boot 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6836/2217260833,v1:172.21.15.110:6838/2217260833] up:boot 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6834/2253059255,v1:172.21.15.110:6837/2253059255] up:boot 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.110:6835/3896928820,v1:172.21.15.110:6839/3896928820] up:boot 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6836/4234467521,v1:172.21.15.131:6839/4234467521] up:boot 2024-04-23T18:27:53.992 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: mds.? [v2:172.21.15.131:6837/2717670530,v1:172.21.15.131:6838/2717670530] up:boot 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: fsmap 12 up:standby 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='mgr.15297 172.21.15.26:0/2233153922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:53.993 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:53 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:54.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.280+0000 7fb679ffb640 1 -- 172.21.15.26:0/2614763009 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) ==== 181+0+0 (secure 0 0 0) 0x7fb66c0bcf10 con 0x7fb67c19d000 2024-04-23T18:27:54.273 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_metadata' removed 2024-04-23T18:27:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 msgr2=0x7fb6680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 0x7fb6680a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb674005fd0 tx=0x7fb674009500 comp rx=0 tx=0).stop 2024-04-23T18:27:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 msgr2=0x7fb67c19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb66c02a930 tx=0x7fb66c003c00 comp rx=0 tx=0).stop 2024-04-23T18:27:54.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 shutdown_connections 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7fb6680a3d20 0x7fb6680a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c106160 0x7fb67c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fb67c19d000 0x7fb67c19d4b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 --2- 172.21.15.26:0/2614763009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb67c101090 0x7fb67c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 >> 172.21.15.26:0/2614763009 conn(0x7fb67c077f50 msgr2=0x7fb67c069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.284+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 shutdown_connections 2024-04-23T18:27:54.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.285+0000 7fb683243640 1 -- 172.21.15.26:0/2614763009 wait complete. 2024-04-23T18:27:54.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T18:27:54.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 -- 172.21.15.26:0/3298612287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c806a840 msgr2=0x7f29c806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:54.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/3298612287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c806a840 0x7f29c806ac20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009900 tx=0x7f29b802a410 comp rx=0 tx=0).stop 2024-04-23T18:27:54.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 -- 172.21.15.26:0/3298612287 shutdown_connections 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/3298612287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c806b1f0 0x7f29c8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/3298612287 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c8107270 0x7f29c810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/3298612287 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c806a840 0x7f29c806ac20 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 -- 172.21.15.26:0/3298612287 >> 172.21.15.26:0/3298612287 conn(0x7f29c8077eb0 msgr2=0x7f29c80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 -- 172.21.15.26:0/3298612287 shutdown_connections 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.418+0000 7f29cd8e1640 1 -- 172.21.15.26:0/3298612287 wait complete. 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 Processor -- start 2024-04-23T18:27:54.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 -- start start 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 0x7f29c819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c81985c0 0x7f29c8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 -- --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_getmap magic: 0 -- 0x7f29c81121e0 con 0x7f29c8107270 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f29c8112060 con 0x7f29c81985c0 2024-04-23T18:27:54.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29cd8e1640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 -- 0x7f29c8112360 con 0x7f29c806a840 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 --2- >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.26:3300/0 says I am v2:172.21.15.26:33164/0 (socket says 172.21.15.26:33164) 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 -- 172.21.15.26:0/4064561905 learned_addr learned my addr 172.21.15.26:0/4064561905 (peer_addr_for_me v2:172.21.15.26:0/0) 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c77fe640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c81985c0 0x7f29c8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c6ffd640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 0x7f29c819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 -- 172.21.15.26:0/4064561905 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 msgr2=0x7f29c819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 0x7f29c819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 -- 172.21.15.26:0/4064561905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c81985c0 msgr2=0x7f29c8198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c81985c0 0x7f29c8198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.419+0000 7f29c67fc640 1 -- 172.21.15.26:0/4064561905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c81990e0 con 0x7f29c8107270 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29c6ffd640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 0x7f29c819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29c67fc640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00d770 tx=0x7f29bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc017070 con 0x7f29c8107270 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29bc009c30 con 0x7f29c8107270 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c81993d0 con 0x7f29c8107270 2024-04-23T18:27:54.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc00f620 con 0x7f29c8107270 2024-04-23T18:27:54.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.420+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c81a5620 con 0x7f29c8107270 2024-04-23T18:27:54.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.421+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c80691c0 con 0x7f29c8107270 2024-04-23T18:27:54.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.422+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f29bc01a070 con 0x7f29c8107270 2024-04-23T18:27:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.423+0000 7f29cc8df640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 0x7f29b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.423+0000 7f29c6ffd640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 0x7f29b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.423+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8202+0+0 (secure 0 0 0) 0x7f29bc0d0290 con 0x7f29c8107270 2024-04-23T18:27:54.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.423+0000 7f29c6ffd640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 0x7f29b00a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b8031400 comp rx=0 tx=0).ready entity=mgr.15297 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:54.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.428+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29bc009de0 con 0x7f29c8107270 2024-04-23T18:27:54.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:54.663+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 --> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f29c8199af0 con 0x7f29c8107270 2024-04-23T18:27:54.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:54 smithi110 ceph-mon[32585]: pgmap v335: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2024-04-23T18:27:54.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:54 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:27:54.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:54 smithi110 ceph-mon[32585]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:27:54.941 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:54 smithi110 ceph-mon[32585]: from='client.? 172.21.15.26:0/4064561905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:54.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:54 smithi026 ceph-mon[29364]: pgmap v335: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2024-04-23T18:27:54.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:54 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:27:54.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:54 smithi026 ceph-mon[29364]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:27:54.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:54 smithi026 ceph-mon[29364]: from='client.? 172.21.15.26:0/4064561905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:54.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:54 smithi131 ceph-mon[33041]: pgmap v335: 97 pgs: 97 active+clean; 358 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2024-04-23T18:27:54.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:54 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/2614763009' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:27:54.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:54 smithi131 ceph-mon[33041]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:27:54.991 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:54 smithi131 ceph-mon[33041]: from='client.? 172.21.15.26:0/4064561905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:27:55.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.281+0000 7f29cc8df640 1 -- 172.21.15.26:0/4064561905 <== mon.0 v2:172.21.15.26:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) ==== 169+0+0 (secure 0 0 0) 0x7f29bc09ca10 con 0x7f29c8107270 2024-04-23T18:27:55.274 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'cephfs_data' removed 2024-04-23T18:27:55.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 msgr2=0x7f29b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 0x7f29b00a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b8031400 comp rx=0 tx=0).stop 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 msgr2=0x7f29c819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00d770 tx=0x7f29bc00dc30 comp rx=0 tx=0).stop 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 shutdown_connections 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:6800/2566950754,v1:172.21.15.26:6801/2566950754] conn(0x7f29b00a3d20 0x7f29b00a61e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29c81985c0 0x7f29c8198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f29c8107270 0x7f29c819e950 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:55.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 --2- 172.21.15.26:0/4064561905 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f29c806a840 0x7f29c819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:55.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 >> 172.21.15.26:0/4064561905 conn(0x7f29c8077eb0 msgr2=0x7f29c80ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:55.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 shutdown_connections 2024-04-23T18:27:55.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-23T18:27:55.285+0000 7f29cd8e1640 1 -- 172.21.15.26:0/4064561905 wait complete. 2024-04-23T18:27:55.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:27:55.326 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:27:55.352 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:27:55.377 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T18:27:55.377 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:27:55.394 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:27:55.408 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:27:55.432 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T18:27:55.432 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T18:27:55.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:27:55.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:55 smithi026 systemd[1]: Stopping Ceph mon.a for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:27:56.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a[29360]: 2024-04-23T18:27:55.759+0000 7f5be3fd9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:27:56.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a[29360]: 2024-04-23T18:27:55.759+0000 7f5be3fd9700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T18:27:56.143 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:55 smithi026 podman[89648]: 2024-04-23 18:27:55.861998354 +0000 UTC m=+0.221449288 container died d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:27:56.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:56 smithi026 podman[89648]: 2024-04-23 18:27:56.249312083 +0000 UTC m=+0.608763044 container cleanup d69906676dfcbd4f1f4ead573ee6e5a7763466b83a3a1719accbe147faa04525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-23T18:27:56.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Apr 23 18:27:56 smithi026 bash[89648]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:27:56.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T18:27:56.717 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:27:56.717 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T18:27:56.718 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T18:27:56.718 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.b 2024-04-23T18:27:57.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:56 smithi110 systemd[1]: Stopping Ceph mon.b for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:27:57.431 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b[32581]: 2024-04-23T18:27:57.029+0000 7fa684a93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:27:57.431 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b[32581]: 2024-04-23T18:27:57.029+0000 7fa684a93700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:27:57.431 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 podman[68761]: 2024-04-23 18:27:57.182983503 +0000 UTC m=+0.258404769 container died cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T18:27:57.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 podman[68761]: 2024-04-23 18:27:57.529522172 +0000 UTC m=+0.604943437 container cleanup cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:27:57.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 bash[68761]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b 2024-04-23T18:27:57.691 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 23 18:27:57 smithi110 podman[68775]: 2024-04-23 18:27:57.673565489 +0000 UTC m=+0.487246401 container remove cb6ebff28cab4aa71043e68d5ed8672f5b96a9308de5a43c19578545bb473fb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:27:57.883 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T18:27:57.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:27:57.929 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T18:27:57.929 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T18:27:57.929 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.c 2024-04-23T18:27:58.230 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:57 smithi131 systemd[1]: Stopping Ceph mon.c for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:27:58.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-c[33037]: 2024-04-23T18:27:58.237+0000 7f4ef7124700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:27:58.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-c[33037]: 2024-04-23T18:27:58.237+0000 7f4ef7124700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:27:58.491 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:58 smithi131 podman[69975]: 2024-04-23 18:27:58.268535291 +0000 UTC m=+0.134200026 container died d4b4b575b5e643b277301b11c785d1e0ef1acdd07a8ef6e1fd779dafbc7530c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:27:58.881 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:58 smithi131 podman[69975]: 2024-04-23 18:27:58.654589452 +0000 UTC m=+0.520254187 container cleanup d4b4b575b5e643b277301b11c785d1e0ef1acdd07a8ef6e1fd779dafbc7530c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T18:27:58.881 INFO:journalctl@ceph.mon.c.smithi131.stdout:Apr 23 18:27:58 smithi131 bash[69975]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mon-c 2024-04-23T18:27:59.023 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T18:27:59.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:27:59.069 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T18:27:59.069 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T18:27:59.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:27:59.387 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 systemd[1]: Stopping Ceph mgr.x for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:27:59.705 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:27:59.394+0000 7f7bb0e0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:27:59.705 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x[29594]: 2024-04-23T18:27:59.394+0000 7f7bb0e0f700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-23T18:27:59.705 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 podman[89770]: 2024-04-23 18:27:59.42670431 +0000 UTC m=+0.154536230 container died 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:27:59.958 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 podman[89770]: 2024-04-23 18:27:59.812346018 +0000 UTC m=+0.540177929 container cleanup 0403e4f3d07063f90c61d99470da197c1ffaf523bfa982ce2d21e5bb8e07a789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:27:59.958 INFO:journalctl@ceph.mgr.x.smithi026.stdout:Apr 23 18:27:59 smithi026 bash[89770]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-x 2024-04-23T18:28:00.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T18:28:00.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:00.270 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T18:28:00.270 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T18:28:00.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.y 2024-04-23T18:28:00.606 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:28:00 smithi110 systemd[1]: Stopping Ceph mgr.y for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:00.936 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:28:00 smithi110 podman[68884]: 2024-04-23 18:28:00.614832458 +0000 UTC m=+0.147808732 container died dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T18:28:01.191 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:28:01 smithi110 podman[68884]: 2024-04-23 18:28:01.018873984 +0000 UTC m=+0.551850258 container cleanup dffa7376cdd3035865a3eef2be71bd1e6facf3ac8dc8359bc1b578dedd16d615 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:28:01.192 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 23 18:28:01 smithi110 bash[68884]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-y 2024-04-23T18:28:01.333 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T18:28:01.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:01.366 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T18:28:01.366 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-23T18:28:01.366 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.z 2024-04-23T18:28:01.688 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:28:01 smithi131 systemd[1]: Stopping Ceph mgr.z for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:01.985 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:28:01 smithi131 podman[70100]: 2024-04-23 18:28:01.696570683 +0000 UTC m=+0.127067624 container died 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:28:02.241 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:28:02 smithi131 podman[70100]: 2024-04-23 18:28:02.067131146 +0000 UTC m=+0.497628086 container cleanup 4fc2e7ecaeb902097b6ecbd604cebca610195a303b9582f68090b6c9f38dc6e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:28:02.241 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Apr 23 18:28:02 smithi131 bash[70100]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mgr-z 2024-04-23T18:28:02.468 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-23T18:28:02.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:02.502 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-23T18:28:02.503 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T18:28:02.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.0 2024-04-23T18:28:02.801 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:02 smithi026 systemd[1]: Stopping Ceph osd.0 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:03.207 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0[41609]: 2024-04-23T18:28:02.807+0000 7f47d812e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:03.208 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0[41609]: 2024-04-23T18:28:02.807+0000 7f47d812e700 -1 osd.0 101 *** Got signal Terminated *** 2024-04-23T18:28:03.208 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0[41609]: 2024-04-23T18:28:02.807+0000 7f47d812e700 -1 osd.0 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:08.458 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:08 smithi026 podman[89891]: 2024-04-23 18:28:08.184969104 +0000 UTC m=+5.475163414 container died 3c1b9ad246c4d2d24e6371da33520319c4bcf40d22095fbe9924da8938e0231c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:28:08.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:08 smithi026 podman[89891]: 2024-04-23 18:28:08.666711138 +0000 UTC m=+5.956905453 container cleanup 3c1b9ad246c4d2d24e6371da33520319c4bcf40d22095fbe9924da8938e0231c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-23T18:28:08.772 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:08 smithi026 bash[89891]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0 2024-04-23T18:28:09.208 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:08 smithi026 podman[89908]: 2024-04-23 18:28:08.943252566 +0000 UTC m=+0.752651468 container remove 3c1b9ad246c4d2d24e6371da33520319c4bcf40d22095fbe9924da8938e0231c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:09.580 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.184890985 +0000 UTC m=+0.021550232 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:09.580 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.327459522 +0000 UTC m=+0.164118743 container create 04afb4e46f77405e86c682e158aad31bb5e139f9649c06075bd74e4b84e1f22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T18:28:09.580 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.58805177 +0000 UTC m=+0.424710994 container init 04afb4e46f77405e86c682e158aad31bb5e139f9649c06075bd74e4b84e1f22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T18:28:09.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.591585941 +0000 UTC m=+0.428245170 container start 04afb4e46f77405e86c682e158aad31bb5e139f9649c06075bd74e4b84e1f22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:09.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.796749165 +0000 UTC m=+0.633408393 container attach 04afb4e46f77405e86c682e158aad31bb5e139f9649c06075bd74e4b84e1f22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416) 2024-04-23T18:28:10.199 INFO:journalctl@ceph.osd.0.smithi026.stdout:Apr 23 18:28:09 smithi026 podman[90006]: 2024-04-23 18:28:09.899090408 +0000 UTC m=+0.735749633 container died 04afb4e46f77405e86c682e158aad31bb5e139f9649c06075bd74e4b84e1f22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T18:28:10.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T18:28:10.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:10.609 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T18:28:10.609 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T18:28:10.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.1 2024-04-23T18:28:10.924 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:10 smithi110 systemd[1]: Stopping Ceph osd.1 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:11.191 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1[37156]: 2024-04-23T18:28:10.930+0000 7faf92f8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:11.191 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1[37156]: 2024-04-23T18:28:10.930+0000 7faf92f8a700 -1 osd.1 101 *** Got signal Terminated *** 2024-04-23T18:28:11.191 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1[37156]: 2024-04-23T18:28:10.930+0000 7faf92f8a700 -1 osd.1 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:16.591 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:16 smithi110 podman[69009]: 2024-04-23 18:28:16.307616147 +0000 UTC m=+5.503505714 container died 32861525d3804ff57d167db2b95b6eb09bd55e18948aa0772f56913f50cf3e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:28:16.899 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:16 smithi110 podman[69009]: 2024-04-23 18:28:16.765019995 +0000 UTC m=+5.960909569 container cleanup 32861525d3804ff57d167db2b95b6eb09bd55e18948aa0772f56913f50cf3e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T18:28:16.899 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:16 smithi110 bash[69009]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1 2024-04-23T18:28:17.191 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:16 smithi110 podman[69026]: 2024-04-23 18:28:16.90759573 +0000 UTC m=+0.596306039 container remove 32861525d3804ff57d167db2b95b6eb09bd55e18948aa0772f56913f50cf3e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:17.619 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.206268019 +0000 UTC m=+0.018111813 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:17.619 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.316799668 +0000 UTC m=+0.128643495 container create 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:17.619 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.527232278 +0000 UTC m=+0.339076058 container init 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:28:17.619 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.530247341 +0000 UTC m=+0.342091128 container start 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T18:28:17.620 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.627196139 +0000 UTC m=+0.439039939 container attach 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:28:17.941 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:17 smithi110 podman[69122]: 2024-04-23 18:28:17.822061404 +0000 UTC m=+0.633905185 container died 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:28:18.398 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 23 18:28:18 smithi110 podman[69149]: 2024-04-23 18:28:18.389213151 +0000 UTC m=+0.558545972 container remove 36ae0b28de6ac7bc74da8a40d0b17a75d72d9743d3524d0cc633df26e013d917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:28:18.404 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T18:28:18.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:18.441 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T18:28:18.441 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T18:28:18.442 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.2 2024-04-23T18:28:18.741 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:18 smithi131 systemd[1]: Stopping Ceph osd.2 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:19.006 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2[38188]: 2024-04-23T18:28:18.761+0000 7f2267cfd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:19.007 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2[38188]: 2024-04-23T18:28:18.761+0000 7f2267cfd700 -1 osd.2 101 *** Got signal Terminated *** 2024-04-23T18:28:19.007 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2[38188]: 2024-04-23T18:28:18.761+0000 7f2267cfd700 -1 osd.2 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:24.614 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:24 smithi131 podman[70222]: 2024-04-23 18:28:24.328779589 +0000 UTC m=+5.687511320 container died 214b83086729e40f40916859bb26a0f06cab568c09b57c94b49899371e71fe39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T18:28:24.939 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:24 smithi131 podman[70222]: 2024-04-23 18:28:24.720829327 +0000 UTC m=+6.079561067 container cleanup 214b83086729e40f40916859bb26a0f06cab568c09b57c94b49899371e71fe39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:28:24.939 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:24 smithi131 bash[70222]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2 2024-04-23T18:28:25.232 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:24 smithi131 podman[70250]: 2024-04-23 18:28:24.947180341 +0000 UTC m=+0.617803726 container remove 214b83086729e40f40916859bb26a0f06cab568c09b57c94b49899371e71fe39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T18:28:25.491 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.143870902 +0000 UTC m=+0.018340337 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:25.492 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.25645454 +0000 UTC m=+0.130923956 container create 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-23T18:28:25.492 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.467058849 +0000 UTC m=+0.341528271 container init 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T18:28:25.492 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.47008042 +0000 UTC m=+0.344549837 container start 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-04-23T18:28:25.751 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.541866442 +0000 UTC m=+0.416335875 container attach 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T18:28:26.086 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:25 smithi131 podman[70339]: 2024-04-23 18:28:25.760274843 +0000 UTC m=+0.634744253 container died 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:28:26.337 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:26 smithi131 podman[70365]: 2024-04-23 18:28:26.328861323 +0000 UTC m=+0.560005898 container remove 3a38709aeaf5fa7225f085dfc0c89eb3840f1d2334e4c15b0cdb1e33e95b773f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:28:26.338 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:26 smithi131 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-23T18:28:26.338 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:26 smithi131 systemd[1]: Stopped Ceph osd.2 for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:28:26.338 INFO:journalctl@ceph.osd.2.smithi131.stdout:Apr 23 18:28:26 smithi131 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.2.service: Consumed 40.596s CPU time. 2024-04-23T18:28:26.339 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T18:28:26.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:26.378 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T18:28:26.378 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T18:28:26.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.3 2024-04-23T18:28:26.686 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:26 smithi026 systemd[1]: Stopping Ceph osd.3 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:26.957 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:28:26.693+0000 7f57dbf7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:26.958 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:28:26.693+0000 7f57dbf7f700 -1 osd.3 101 *** Got signal Terminated *** 2024-04-23T18:28:26.958 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:26 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:28:26.693+0000 7f57dbf7f700 -1 osd.3 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:29.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:29 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:29.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:30.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:29 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:29.943+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:30.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:29.799+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:30.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:30 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:30.073+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:31.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:30 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:30.921+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:31.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:30.832+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:31.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:30.822+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:31.458 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:31 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3[48148]: 2024-04-23T18:28:31.122+0000 7f57d8573700 -1 osd.3 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.229768+0000 front 2024-04-23T18:28:07.229712+0000 (oldest deadline 2024-04-23T18:28:30.129857+0000) 2024-04-23T18:28:31.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:31 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:31.079+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:32.078 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:31 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:31.744+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:32.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:31 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:31.915+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:32.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:31.850+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:32.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:31.850+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:32.395 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:32 smithi026 podman[90114]: 2024-04-23 18:28:32.085998581 +0000 UTC m=+5.495523630 container died 9af49edcd13c1747d3bd5a28ce8cb3b8420120ba0925e1b0697a4e06805d3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T18:28:32.395 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:32 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:32.124+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:32.708 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:32 smithi026 podman[90114]: 2024-04-23 18:28:32.502277422 +0000 UTC m=+5.911802475 container cleanup 9af49edcd13c1747d3bd5a28ce8cb3b8420120ba0925e1b0697a4e06805d3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:32.708 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:32 smithi026 bash[90114]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3 2024-04-23T18:28:33.073 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:32 smithi026 podman[90131]: 2024-04-23 18:28:32.736836108 +0000 UTC m=+0.643249909 container remove 9af49edcd13c1747d3bd5a28ce8cb3b8420120ba0925e1b0697a4e06805d3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:28:33.074 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:32 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:32.778+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:33.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:32 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:32.939+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:33.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:32.888+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:33.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:32.860+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:33.378 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:33 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:33.087+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:33.378 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:32.984631179 +0000 UTC m=+0.018510600 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:33.378 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:33.095857727 +0000 UTC m=+0.129737133 container create b4d808c59c9c6da43b981391a7172339224679c8caa1b47e0a366ab183478dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:33.690 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:33.386455402 +0000 UTC m=+0.420334812 container init b4d808c59c9c6da43b981391a7172339224679c8caa1b47e0a366ab183478dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T18:28:33.691 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:33.390142582 +0000 UTC m=+0.424021988 container start b4d808c59c9c6da43b981391a7172339224679c8caa1b47e0a366ab183478dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T18:28:33.691 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:33.481602847 +0000 UTC m=+0.515482256 container attach b4d808c59c9c6da43b981391a7172339224679c8caa1b47e0a366ab183478dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:28:33.958 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:33 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:33.802+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:33.959 INFO:journalctl@ceph.osd.3.smithi026.stdout:Apr 23 18:28:33 smithi026 podman[90220]: 2024-04-23 18:28:33.69885285 +0000 UTC m=+0.732732274 container died b4d808c59c9c6da43b981391a7172339224679c8caa1b47e0a366ab183478dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T18:28:34.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:33.857+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:33.899+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:34.419 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:34 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:34.116+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:34.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T18:28:34.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:33 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:33.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:34.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:34.487 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T18:28:34.487 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T18:28:34.488 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.4 2024-04-23T18:28:34.768 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:34 smithi110 systemd[1]: Stopping Ceph osd.4 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:35.086 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:34 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:34.793+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:35.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:34 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:34.987+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:35.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:34 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:34.774+0000 7f3381b77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:35.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:34 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:34.774+0000 7f3381b77700 -1 osd.4 101 *** Got signal Terminated *** 2024-04-23T18:28:35.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:34 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:34.774+0000 7f3381b77700 -1 osd.4 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:35.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:34 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:34.873+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:35.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:34 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:34.923+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:35.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:35 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:35.092+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:35.957 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:35 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:35.818+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:36.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:35 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:35.948+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:36.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:35 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:35.849+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:36.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:35 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:35.957+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:36.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:36 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:36.049+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:37.042 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:36 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:36.788+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:37.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:36 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:36.821+0000 7f337a163700 -1 osd.4 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.617673+0000 front 2024-04-23T18:28:15.617775+0000 (oldest deadline 2024-04-23T18:28:36.717815+0000) 2024-04-23T18:28:37.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:36 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:36.990+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:37.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:36 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:36.812+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:37.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:36 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:36.965+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:37.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:37.050+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:37.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:37.050+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:38.083 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:37.831+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:38.083 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:37.831+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:38.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:37 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:37.778+0000 7f337a163700 -1 osd.4 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.617673+0000 front 2024-04-23T18:28:15.617775+0000 (oldest deadline 2024-04-23T18:28:36.717815+0000) 2024-04-23T18:28:38.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:37 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:37.979+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:38.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:37 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:37.979+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:38.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:37 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:37.774+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:38.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:37 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:37.947+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:38.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:38.090+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:38.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:38.090+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:39.067 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:38.801+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:39.067 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:38 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:38.801+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:39.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:38 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:38.802+0000 7f337a163700 -1 osd.4 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.617673+0000 front 2024-04-23T18:28:15.617775+0000 (oldest deadline 2024-04-23T18:28:36.717815+0000) 2024-04-23T18:28:39.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:39.013+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:39.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:39.013+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:39.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:38 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:38.800+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:39.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:38 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:38.800+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:39.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:38 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:38.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:39.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:39.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:39.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:39.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:39.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:39.264+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:40.111 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:39.824+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:40.112 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:39 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:39.824+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:40.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4[43834]: 2024-04-23T18:28:39.753+0000 7f337a163700 -1 osd.4 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.617673+0000 front 2024-04-23T18:28:15.617775+0000 (oldest deadline 2024-04-23T18:28:36.717815+0000) 2024-04-23T18:28:40.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:39.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:40.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:39 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:39.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:40.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:39 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:39.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:40.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:39 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:39.821+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:40.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:39 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:39.821+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:40.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:40.119+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:40.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:40.119+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:40.609 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:40 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:40.215+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:40.610 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:40 smithi110 podman[69243]: 2024-04-23 18:28:40.330916057 +0000 UTC m=+5.647026363 container died 75f5d13b7690500d078cc634571af22b877996af506e4b831d5a9ea10ba8d413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T18:28:40.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:40 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:40.452+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:40.902 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:40 smithi110 podman[69243]: 2024-04-23 18:28:40.725737568 +0000 UTC m=+6.041847859 container cleanup 75f5d13b7690500d078cc634571af22b877996af506e4b831d5a9ea10ba8d413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-23T18:28:40.902 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:40 smithi110 bash[69243]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4 2024-04-23T18:28:41.191 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:40 smithi110 podman[69262]: 2024-04-23 18:28:40.910108405 +0000 UTC m=+0.574539231 container remove 75f5d13b7690500d078cc634571af22b877996af506e4b831d5a9ea10ba8d413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:28:41.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:40 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:40.956+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:41.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:40 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:40.956+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:41.207 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:40.847+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:41.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:40 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:40.847+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:41.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:41.097+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:41.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:41.097+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:41.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:41.000+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:41.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:41.000+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:41.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:40 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:40.798+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:41.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:40 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:40.798+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:41.522 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:41 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:41.218+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:41.523 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.138425976 +0000 UTC m=+0.017953149 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:41.523 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.261202434 +0000 UTC m=+0.140729597 container create 108be69d5f402d4de50fe2817762465ac67767022c1202f1189cc0a5de83ccb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:28:41.523 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.455471148 +0000 UTC m=+0.334998313 container init 108be69d5f402d4de50fe2817762465ac67767022c1202f1189cc0a5de83ccb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:28:41.523 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.458634474 +0000 UTC m=+0.338161639 container start 108be69d5f402d4de50fe2817762465ac67767022c1202f1189cc0a5de83ccb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T18:28:41.523 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.530504658 +0000 UTC m=+0.410031834 container attach 108be69d5f402d4de50fe2817762465ac67767022c1202f1189cc0a5de83ccb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T18:28:41.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:41.415+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:41.910 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 23 18:28:41 smithi110 podman[69354]: 2024-04-23 18:28:41.744583026 +0000 UTC m=+0.624110191 container died 108be69d5f402d4de50fe2817762465ac67767022c1202f1189cc0a5de83ccb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:28:42.068 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:41.805+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:42.068 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:41 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:41.805+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:42.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:41 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:41.917+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:42.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:41 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:41.917+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:42.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:41.990+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:42.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:41.990+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:42.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:41.845+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:42.242 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:41 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:41.845+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:42.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T18:28:42.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:42.352 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T18:28:42.352 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T18:28:42.352 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.5 2024-04-23T18:28:42.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:42.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:42.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:42.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:42.665 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 systemd[1]: Stopping Ceph osd.5 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:42.666 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:42.407+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:42.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:42 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:42.205+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:42.991 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:42.672+0000 7f51be1d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:42.991 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:42.672+0000 7f51be1d3700 -1 osd.5 101 *** Got signal Terminated *** 2024-04-23T18:28:42.991 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:42.672+0000 7f51be1d3700 -1 osd.5 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:42.991 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:42.850+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:42.991 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:42 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:42.850+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:43.057 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:42.788+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:43.058 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:42.788+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:43.160 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:42 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:42.870+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:43.160 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:42 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:42.870+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:43.368 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:43.039+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:43.368 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:43.039+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:43.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:43 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:43.168+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:43.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:43.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:43.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:43.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:43.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:43.375+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:44.141 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:43 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:43.834+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:44.142 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:43 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:43.834+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:44.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:43.821+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:44.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:43 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:43.821+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:44.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:44.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:44.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:44.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:44.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:43.827+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:44.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:43 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:43.827+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:44.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:44 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:44.047+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:44.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:44 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:44.047+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:44.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:44 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:44.149+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:44.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:44 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:44.412+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:45.180 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:44 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:44.884+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:45.180 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:44 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:44.884+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:45.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:44.866+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:45.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:44 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:44.866+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:45.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:45.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:45.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:45.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:45.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:45.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:45.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:44 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:44.844+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:45.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:44 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:44.844+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:45.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:45.093+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:45.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:45.093+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:45.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:45 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:45.187+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:45.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:45.425+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:45.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:45.425+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:46.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:45 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:45.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:46.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:45 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:45.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:45.911+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:45.911+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:45 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:45.911+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:46.053+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:46.053+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:46.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:46.053+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:46.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:46.048+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:46.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:46.048+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:46.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:45.810+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:46.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:45.810+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:46.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:45 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:45.810+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.273323+0000 front 2024-04-23T18:28:23.273357+0000 (oldest deadline 2024-04-23T18:28:44.973435+0000) 2024-04-23T18:28:46.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:46 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:46.226+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:46.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:46 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:46.226+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:46.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:46.448+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:46.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:46.448+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:47.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:46.903+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:47.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:46.903+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:47.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:46 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:46.903+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:47.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:47.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:47.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:47.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:47.209 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:47.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:47.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:46.856+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:07.471546+0000 front 2024-04-23T18:28:07.471650+0000 (oldest deadline 2024-04-23T18:28:30.371452+0000) 2024-04-23T18:28:47.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:46.856+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.072727+0000 front 2024-04-23T18:28:12.072490+0000 (oldest deadline 2024-04-23T18:28:37.972469+0000) 2024-04-23T18:28:47.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:46 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5[44712]: 2024-04-23T18:28:46.856+0000 7f51b67bf700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.273323+0000 front 2024-04-23T18:28:23.273357+0000 (oldest deadline 2024-04-23T18:28:44.973435+0000) 2024-04-23T18:28:47.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:47 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:47.046+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:47.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:47 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:47.046+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:47.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:47 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:47.046+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:47.247 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:46 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:46.957+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:47.247 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:46 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:46.957+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:47.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:47 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:47.255+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:47.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:47 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:47.255+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:47.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:47 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:47.415+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:47.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:47 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:47.415+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:48.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:48.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:48.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:47.888+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:47.888+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:48.139 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:47.888+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:48.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:47 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:47.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:48.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:47 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:47.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:48.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:47 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:47.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:48.291 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 podman[70445]: 2024-04-23 18:28:48.025879187 +0000 UTC m=+5.473558039 container died 18ac30b513fd4e01a8224039ed71bb977649e515b9c2d90ac876dc55c33c6b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T18:28:48.291 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.016+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:48.291 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.016+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:48.291 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.016+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:48.624 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 podman[70445]: 2024-04-23 18:28:48.406084856 +0000 UTC m=+5.853763706 container cleanup 18ac30b513fd4e01a8224039ed71bb977649e515b9c2d90ac876dc55c33c6b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T18:28:48.624 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 bash[70445]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5 2024-04-23T18:28:48.624 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:48.429+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:48.624 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:48.429+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:48.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:48.270+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:48.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:48.270+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:48.942 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 podman[70462]: 2024-04-23 18:28:48.632080441 +0000 UTC m=+0.599948034 container remove 18ac30b513fd4e01a8224039ed71bb977649e515b9c2d90ac876dc55c33c6b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T18:28:49.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:48.899+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:49.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:48.899+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:49.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:48 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:48.899+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:49.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:49.077+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:49.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:49.077+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:49.209 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:49.077+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:49.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.982+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:49.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.982+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:49.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:48 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:48.982+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:49.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 podman[70553]: 2024-04-23 18:28:48.852631263 +0000 UTC m=+0.017987310 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:49.241 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:48 smithi131 podman[70553]: 2024-04-23 18:28:48.949893518 +0000 UTC m=+0.115249607 container create 0ecdbd2f54f177ac0021159bb2fbb97f514c98ebceaf346f289271c69d1e4229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:28:49.242 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:49 smithi131 podman[70553]: 2024-04-23 18:28:49.193647051 +0000 UTC m=+0.359003102 container init 0ecdbd2f54f177ac0021159bb2fbb97f514c98ebceaf346f289271c69d1e4229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T18:28:49.242 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:49 smithi131 podman[70553]: 2024-04-23 18:28:49.196800213 +0000 UTC m=+0.362156260 container start 0ecdbd2f54f177ac0021159bb2fbb97f514c98ebceaf346f289271c69d1e4229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-23T18:28:49.286 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:48.954+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:49.287 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:48.954+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:49.287 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:48 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:48.954+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:49.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:49.294+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:49.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:49.294+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:49.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:49 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:49.473+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:49.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:49 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:49.473+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:49.742 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:49 smithi131 podman[70553]: 2024-04-23 18:28:49.251874545 +0000 UTC m=+0.417230607 container attach 0ecdbd2f54f177ac0021159bb2fbb97f514c98ebceaf346f289271c69d1e4229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:28:49.742 INFO:journalctl@ceph.osd.5.smithi131.stdout:Apr 23 18:28:49 smithi131 podman[70553]: 2024-04-23 18:28:49.487816887 +0000 UTC m=+0.653172945 container died 0ecdbd2f54f177ac0021159bb2fbb97f514c98ebceaf346f289271c69d1e4229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-23T18:28:50.065 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T18:28:50.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:50.103 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T18:28:50.103 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T18:28:50.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.6 2024-04-23T18:28:50.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:49.928+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:50.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:49.928+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:50.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:49 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:49.928+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:50.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:50.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:50.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:49.935+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:49.935+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:50.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:49 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:49.935+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:50.209 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 systemd[1]: Stopping Ceph osd.6 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:50.315 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.011+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:50.315 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.011+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:50.315 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.011+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:50.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:50 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:50.325+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:50.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:50 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:50.325+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:50.708 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.420+0000 7f813bab5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:50.708 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.420+0000 7f813bab5700 -1 osd.6 101 *** Got signal Terminated *** 2024-04-23T18:28:50.708 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.420+0000 7f813bab5700 -1 osd.6 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:50.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:50.485+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:50.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:50.485+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:51.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:50 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:50.927+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:51.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:50 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:50.927+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:51.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:50 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:50.927+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:51.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.998+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:51.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.998+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:51.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:50 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:50.998+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.968+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.968+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:50 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:50.968+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:51.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:51.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:51.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:51.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:51.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:51 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:51.309+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:51.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:51 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:51.309+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:51.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:51 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:51.473+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:51.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:51 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:51.473+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:52.327 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:51 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:51.948+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:52.327 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:51 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:51.948+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:52.327 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:51 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:51.948+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:52.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:51.996+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:52.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:51.996+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:52.458 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:51 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:51.996+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:52.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:52.034+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:52.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:52.034+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:52.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:52.034+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:52.470 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.006+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:52.470 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.006+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:52.470 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.006+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:52.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:52.334+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:52.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:52.334+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:52.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:52.334+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:52.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:52.478+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:52.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:52.478+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:53.207 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:53.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:53.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:53.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:53.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:53.075+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:53.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:52.958+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:53.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:52.958+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:53.209 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:52.958+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:53.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.957+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:53.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.957+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:53.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:52 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:52.957+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:53.370 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:52.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:53.370 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:52.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:53.371 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:52 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:52.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:53.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:53.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:53.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:53.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:53.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:53.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:53.740 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:53 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:53.486+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:53.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:53 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:53.486+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:54.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:53.945+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:54.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:53.945+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:54.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:53 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:53.945+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:54.207 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:54.115+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:54.115+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:54.115+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:53.943+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:53.943+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:53.943+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:54.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:53 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:53.943+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.794429+0000 front 2024-04-23T18:28:27.794378+0000 (oldest deadline 2024-04-23T18:28:53.694465+0000) 2024-04-23T18:28:54.240 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:53 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:53.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:54.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:53 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:53.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:54.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:53 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:53.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:54.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:54.419+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:54.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:54.419+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:54.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:54.419+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:54.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:54.453+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:54.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:54.453+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:54.742 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:54.453+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:28:55.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:54.917+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.790326+0000 front 2024-04-23T18:28:06.790566+0000 (oldest deadline 2024-04-23T18:28:31.490501+0000) 2024-04-23T18:28:55.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:54.917+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:11.491188+0000 front 2024-04-23T18:28:11.491256+0000 (oldest deadline 2024-04-23T18:28:36.791041+0000) 2024-04-23T18:28:55.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:54.917+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.193114+0000 front 2024-04-23T18:28:20.193345+0000 (oldest deadline 2024-04-23T18:28:45.493195+0000) 2024-04-23T18:28:55.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:54 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6[55177]: 2024-04-23T18:28:54.917+0000 7f81380a9700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.794429+0000 front 2024-04-23T18:28:27.794378+0000 (oldest deadline 2024-04-23T18:28:53.694465+0000) 2024-04-23T18:28:55.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:55.087+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:55.209 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:55.087+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:55.209 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:55.087+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:55.209 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:55 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:55.087+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:28:55.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:54.908+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:55.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:54.908+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:55.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:54 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:54.908+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:55.399 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:54.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:55.399 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:54.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:55.400 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:54 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:54.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:55.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:55.407+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:55.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:55.407+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:55.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:55.407+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:55.740 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:55.421+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:55.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:55.421+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:55.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:55.421+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:28:56.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:55.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:56.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:55.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:56.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:55 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:55.946+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:56.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:55.897+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:56.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:55.897+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:56.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:55 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:55.897+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:56.254 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:55 smithi026 podman[90338]: 2024-04-23 18:28:55.966881817 +0000 UTC m=+5.652038212 container died 017740441c30cdfaac7ec64edd6b343cdc2e651d1fb6d34abe548c855d63a77f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:28:56.255 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:56 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:56.059+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:56.255 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:56 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:56.059+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:56.255 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:56 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:56.059+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:56.255 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:56 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:56.059+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:28:56.588 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:56 smithi026 podman[90338]: 2024-04-23 18:28:56.345069814 +0000 UTC m=+6.030226211 container cleanup 017740441c30cdfaac7ec64edd6b343cdc2e651d1fb6d34abe548c855d63a77f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:28:56.588 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:56 smithi026 bash[90338]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6 2024-04-23T18:28:56.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:56.397+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:56.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:56.397+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:56.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:56.397+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:56.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:56.446+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:56.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:56.446+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:56.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:56.446+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:28:56.924 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:56 smithi026 podman[90355]: 2024-04-23 18:28:56.596221467 +0000 UTC m=+0.630746800 container remove 017740441c30cdfaac7ec64edd6b343cdc2e651d1fb6d34abe548c855d63a77f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T18:28:57.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:56.937+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:57.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:56.937+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:57.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:56 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:56.937+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:57.207 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:57.024+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:57.207 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:57.024+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:57.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:57.024+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:57.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:57 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:57.024+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:28:57.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:56 smithi026 podman[90447]: 2024-04-23 18:28:56.836324386 +0000 UTC m=+0.018674391 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:28:57.208 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:56 smithi026 podman[90447]: 2024-04-23 18:28:56.971857014 +0000 UTC m=+0.154207007 container create 14ff76dd55f5356ce540b47aa42e33873bd07711b7e6cf34835693e2e9b2f3a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-23T18:28:57.240 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:56.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:57.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:56.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:57.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:56 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:56.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:57.514 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:57 smithi026 podman[90447]: 2024-04-23 18:28:57.21603057 +0000 UTC m=+0.398380561 container init 14ff76dd55f5356ce540b47aa42e33873bd07711b7e6cf34835693e2e9b2f3a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T18:28:57.514 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:57 smithi026 podman[90447]: 2024-04-23 18:28:57.219331049 +0000 UTC m=+0.401681039 container start 14ff76dd55f5356ce540b47aa42e33873bd07711b7e6cf34835693e2e9b2f3a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:28:57.514 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:57 smithi026 podman[90447]: 2024-04-23 18:28:57.307672119 +0000 UTC m=+0.490022111 container attach 14ff76dd55f5356ce540b47aa42e33873bd07711b7e6cf34835693e2e9b2f3a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T18:28:57.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:57.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:57.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:57.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:57.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:57.378+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:57.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:57.441+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:57.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:57.441+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:57.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:57.441+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:28:57.811 INFO:journalctl@ceph.osd.6.smithi026.stdout:Apr 23 18:28:57 smithi026 podman[90447]: 2024-04-23 18:28:57.522353995 +0000 UTC m=+0.704703990 container died 14ff76dd55f5356ce540b47aa42e33873bd07711b7e6cf34835693e2e9b2f3a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:28:58.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T18:28:58.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:28:58.183 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T18:28:58.183 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T18:28:58.183 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.7 2024-04-23T18:28:58.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:57.913+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:58.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:57.913+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:58.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:57.913+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:58.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:57.913+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:28:58.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:57.828+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:58.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:57.828+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:58.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:57.828+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:58.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:57 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:57.828+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:28:58.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:58 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:58.070+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:58.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:58 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:58.070+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:58.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:58 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:58.070+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:58.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:58 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:58.070+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:28:58.479 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 systemd[1]: Stopping Ceph osd.7 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.351+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.351+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.351+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.487+0000 7f0863b1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.487+0000 7f0863b1b700 -1 osd.7 101 *** Got signal Terminated *** 2024-04-23T18:28:58.480 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:58.487+0000 7f0863b1b700 -1 osd.7 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:28:58.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:58.447+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:58.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:58.447+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:58.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:58.447+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:28:59.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:58.872+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:28:59.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:58.872+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:28:59.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:58.872+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:28:59.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:58 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:58.872+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:28:59.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:58.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:28:59.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:58.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:28:59.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:58.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:28:59.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:58 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:58.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:28:59.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:59.061+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:28:59.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:59.061+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:28:59.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:59.061+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:28:59.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:28:59 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:28:59.061+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:28:59.465 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:59.357+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:28:59.466 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:59.357+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:28:59.466 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:28:59.357+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:28:59.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:59.457+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:28:59.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:59.457+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:28:59.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:28:59.457+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:00.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:59.915+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:00.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:59.915+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:00.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:59.915+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:00.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:28:59 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:28:59.915+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:00.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:59.853+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:00.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:59.853+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:00.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:59.853+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:00.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:28:59 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:28:59.853+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:00.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:00.016+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:00.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:00.016+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:00.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:00.016+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:00.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:00 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:00.016+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:00.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:00.405+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:29:00.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:00.405+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:29:00.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:00.405+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:29:00.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:00.471+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:00.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:00.471+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:00.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:00.471+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:01.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:00.943+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:01.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:00.943+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:01.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:00.943+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:01.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:00 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:00.943+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:01.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:00.858+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:01.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:00.858+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:01.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:00.858+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:01.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:00 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:00.858+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:01.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:01 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:01.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:01.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:01 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:01.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:01.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:01 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:01.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:01.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:01 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:01.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:01.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:01 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:01.041+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:01.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:01.398+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:29:01.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:01.398+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:29:01.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:01.398+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:29:01.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:01.479+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:01.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:01.479+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:01.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:01.479+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:02.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:01.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:02.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:01.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:02.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:01.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:02.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:01.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:02.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:01 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:01.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:02.436 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:01.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:02.436 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:01.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:02.436 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:01.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:02.436 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:01.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:02.436 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:01.981+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:02.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:02.083+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:02.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:02.083+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:02.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:02.083+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:02.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:02.083+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:02.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:02 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:02.083+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:02.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:02.443+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:29:02.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:02.443+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:29:02.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:02.443+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:29:02.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:02.443+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:36.415322+0000 front 2024-04-23T18:28:36.415347+0000 (oldest deadline 2024-04-23T18:29:01.715430+0000) 2024-04-23T18:29:02.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:02.458+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:02.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:02.458+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:02.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:02.458+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:03.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:02.911+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:03.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:02.911+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:03.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:02.911+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:03.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:02.911+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:03.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:02 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:02.911+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:03.440 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:02.951+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:03.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:02.951+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:03.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:02.951+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:03.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:02.951+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:03.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:02 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:02.951+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:03.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:03.093+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:03.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:03.093+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:03.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:03.093+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:03.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:03.093+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:03.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:03 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:03.093+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:03.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:03.448+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.311463+0000 front 2024-04-23T18:28:14.311586+0000 (oldest deadline 2024-04-23T18:28:38.411568+0000) 2024-04-23T18:29:03.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:03.448+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.613434+0000 front 2024-04-23T18:28:20.613551+0000 (oldest deadline 2024-04-23T18:28:45.913333+0000) 2024-04-23T18:29:03.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:03.448+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:29.414600+0000 front 2024-04-23T18:28:29.414707+0000 (oldest deadline 2024-04-23T18:28:52.314497+0000) 2024-04-23T18:29:03.691 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7[50371]: 2024-04-23T18:29:03.448+0000 7f085c107700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:36.415322+0000 front 2024-04-23T18:28:36.415347+0000 (oldest deadline 2024-04-23T18:29:01.715430+0000) 2024-04-23T18:29:03.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:03.431+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:03.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:03.431+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:03.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:03.431+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:03.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:03.431+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:04.145 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:03 smithi110 podman[69459]: 2024-04-23 18:29:03.837665515 +0000 UTC m=+5.458716180 container died e0ba6e9833561968edcc515eb16b4a99a13728cc8fb5f74fd67591ea70c0ce0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:29:04.146 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:03.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:04.146 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:03.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:04.146 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:03.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:04.146 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:03.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:04.146 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:03 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:03.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:04.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:03.937+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:04.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:03.937+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:04.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:03.937+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:04.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:03.937+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:04.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:03 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:03.937+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:04.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69459]: 2024-04-23 18:29:04.15338306 +0000 UTC m=+5.774433734 container cleanup e0ba6e9833561968edcc515eb16b4a99a13728cc8fb5f74fd67591ea70c0ce0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T18:29:04.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 bash[69459]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7 2024-04-23T18:29:04.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69488]: 2024-04-23 18:29:04.354942102 +0000 UTC m=+0.519914474 container remove e0ba6e9833561968edcc515eb16b4a99a13728cc8fb5f74fd67591ea70c0ce0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:29:04.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:04.085+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:04.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:04.085+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:04.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:04.085+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:04.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:04.085+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:04.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:04 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:04.085+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:04.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:04.461+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:04.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:04.461+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:04.742 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:04.461+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:04.742 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:04.461+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:05.036 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:04.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:05.036 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:04.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:05.036 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:04.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:05.036 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:04.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:04 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:04.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69577]: 2024-04-23 18:29:04.668810881 +0000 UTC m=+0.017104332 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69577]: 2024-04-23 18:29:04.781089532 +0000 UTC m=+0.129382959 container create a3fac11d4d0bb0a6b4fb3888b99974ecef0d4a256ed040fafa44088c058aca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69577]: 2024-04-23 18:29:04.958480727 +0000 UTC m=+0.306774163 container init a3fac11d4d0bb0a6b4fb3888b99974ecef0d4a256ed040fafa44088c058aca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:04 smithi110 podman[69577]: 2024-04-23 18:29:04.96182501 +0000 UTC m=+0.310118440 container start a3fac11d4d0bb0a6b4fb3888b99974ecef0d4a256ed040fafa44088c058aca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:05.037 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:05 smithi110 podman[69577]: 2024-04-23 18:29:05.044094487 +0000 UTC m=+0.392387915 container attach a3fac11d4d0bb0a6b4fb3888b99974ecef0d4a256ed040fafa44088c058aca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:29:05.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:04.952+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:05.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:04.952+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:05.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:04.952+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:05.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:04.952+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:05.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:04 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:04.952+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:05.441 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 23 18:29:05 smithi110 podman[69577]: 2024-04-23 18:29:05.253326757 +0000 UTC m=+0.601620202 container died a3fac11d4d0bb0a6b4fb3888b99974ecef0d4a256ed040fafa44088c058aca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:29:05.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:05.057+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:05.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:05.057+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:05.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:05.057+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:05.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:05.057+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:05.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:05 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:05.057+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:05.877 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T18:29:05.898 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:05.500+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:05.898 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:05.500+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:05.898 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:05.500+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:05.898 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:05.501+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:05.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:05.955 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T18:29:05.956 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T18:29:05.956 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.8 2024-04-23T18:29:06.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:05.995+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:06.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:05.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:06.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:05.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:06.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:05.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:06.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:05.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:06.241 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 systemd[1]: Stopping Ceph osd.8 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:06.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:05.905+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:06.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:05.905+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:06.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:05.905+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:06.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:05.905+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:06.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:05 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:05.905+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:06.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:06.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:06.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:06.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:06.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:06.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:06.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:06.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:06.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:06 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:06.078+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.273+0000 7fe1bebc1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.273+0000 7fe1bebc1700 -1 osd.8 101 *** Got signal Terminated *** 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.273+0000 7fe1bebc1700 -1 osd.8 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.472+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.472+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:06.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.472+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:06.742 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:06.472+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:07.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:06.869+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:07.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:06.869+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:07.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:06.869+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:07.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:06.869+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:07.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:06 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:06.869+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:07.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:07.011+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:07.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:07.011+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:07.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:07.011+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:07.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:07.011+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:07.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:07 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:07.011+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:07.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:07.091+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:07.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:07.091+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:07.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:07.091+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:07.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:07.091+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:07.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:07 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:07.091+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:07.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:07.475+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:07.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:07.475+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:07.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:07.475+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:07.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:07.475+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:08.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:07.916+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:08.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:07.916+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:08.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:07.916+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:08.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:07.916+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:08.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:07 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:07.916+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:08.359 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:08.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:08.359 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:08.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:08.359 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:08.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:08.359 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:08.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:08.360 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:08 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:08.062+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:08.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.012+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:08.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.012+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:08.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.012+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:08.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.012+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:08.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.012+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:08.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:08.496+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:08.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:08.496+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:08.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:08.496+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:08.741 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:08.496+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:09.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:09.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:09.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:09.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:09.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:09.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:08 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:08.943+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:09.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.983+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:09.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.983+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:09.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.983+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:09.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.983+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:09.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:08 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:08.983+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:09.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:09.107+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:09.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:09.107+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:09.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:09.107+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:09.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:09.107+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:09.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:09 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:09.107+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:09.927 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:09.530+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:09.927 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:09.530+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:09.928 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:09.530+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:09.928 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:09.530+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:10.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:09 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:09.934+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:10.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.001+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:10.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.001+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:10.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.001+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:10.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.001+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:10.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.001+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:10.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:10 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:10.074+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:10.939 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:10.567+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.478291+0000 front 2024-04-23T18:28:15.478441+0000 (oldest deadline 2024-04-23T18:28:40.178411+0000) 2024-04-23T18:29:10.940 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:10.567+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:23.679308+0000 front 2024-04-23T18:28:23.679376+0000 (oldest deadline 2024-04-23T18:28:44.779366+0000) 2024-04-23T18:29:10.940 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:10.567+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:27.680625+0000 front 2024-04-23T18:28:27.680599+0000 (oldest deadline 2024-04-23T18:28:53.580483+0000) 2024-04-23T18:29:10.940 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8[51246]: 2024-04-23T18:29:10.567+0000 7fe1bb1b5700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.481519+0000 front 2024-04-23T18:28:39.481571+0000 (oldest deadline 2024-04-23T18:29:02.981552+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:11.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:10 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:10.946+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:11.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:11.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:11.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:11.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:11.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:10 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:10.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:11.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:11.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:11.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:11.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:11.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:11.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:11 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:11.051+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:11.947 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:11 smithi131 podman[70671]: 2024-04-23 18:29:11.683020405 +0000 UTC m=+5.521989710 container died 1d57784ee1946d49acc4d21fdbb36f5f25fb40a795d77aed5cadd3b30347eadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:29:12.241 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70671]: 2024-04-23 18:29:12.032208473 +0000 UTC m=+5.871177770 container cleanup 1d57784ee1946d49acc4d21fdbb36f5f25fb40a795d77aed5cadd3b30347eadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:29:12.241 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 bash[70671]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8 2024-04-23T18:29:12.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:12.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:12.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:12.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:12.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:12.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:11 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:11.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:12.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:12.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:12.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:12.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:12.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:12.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:12 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:12.065+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:12.626 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70690]: 2024-04-23 18:29:12.29194586 +0000 UTC m=+0.609483529 container remove 1d57784ee1946d49acc4d21fdbb36f5f25fb40a795d77aed5cadd3b30347eadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:29:12.937 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70782]: 2024-04-23 18:29:12.634250619 +0000 UTC m=+0.101284483 container create c512d008261056c2f7f84e08fd39b598d9256fbf842e726640372327d9b87333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:29:12.937 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70782]: 2024-04-23 18:29:12.550977001 +0000 UTC m=+0.018010888 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:12.937 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70782]: 2024-04-23 18:29:12.869640749 +0000 UTC m=+0.336674613 container init c512d008261056c2f7f84e08fd39b598d9256fbf842e726640372327d9b87333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:29:12.937 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70782]: 2024-04-23 18:29:12.872631079 +0000 UTC m=+0.339664946 container start c512d008261056c2f7f84e08fd39b598d9256fbf842e726640372327d9b87333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-23T18:29:13.241 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:12 smithi131 podman[70782]: 2024-04-23 18:29:12.944689514 +0000 UTC m=+0.411723408 container attach c512d008261056c2f7f84e08fd39b598d9256fbf842e726640372327d9b87333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:29:13.241 INFO:journalctl@ceph.osd.8.smithi131.stdout:Apr 23 18:29:13 smithi131 podman[70782]: 2024-04-23 18:29:13.15957079 +0000 UTC m=+0.626604660 container died c512d008261056c2f7f84e08fd39b598d9256fbf842e726640372327d9b87333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-23T18:29:13.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:13.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:13.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:13.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:13.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:13.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:12 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:12.991+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:13.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:13.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:13.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:13.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:13.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:12 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:12.992+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:13.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:13.104+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:13.771 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T18:29:13.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:13.808 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T18:29:13.808 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T18:29:13.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.9 2024-04-23T18:29:14.105 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:13 smithi026 systemd[1]: Stopping Ceph osd.9 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:14.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:14.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:14.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:14.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:14.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:14.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:13 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:13.976+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:14.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:14.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:14.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:14.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:14.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:14.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:13 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:13.949+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.111+0000 7fdfdcdbc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.111+0000 7fdfdcdbc700 -1 osd.9 101 *** Got signal Terminated *** 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.111+0000 7fdfdcdbc700 -1 osd.9 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:14.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:14.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:14.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:14.459 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:14 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:14.139+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:15.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:15.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:15.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:15.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:15.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:15.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:14 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:14.919+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:15.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:15.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:15.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:15.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:15.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:15.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:14.986+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:15.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:15 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:15.174+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:16.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:16.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:16.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:16.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:16.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:16.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:15 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:15.954+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:16.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:16.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:16.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:16.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:16.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:16.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:15 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:15.964+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:16.708 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:16.708 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:16.708 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:16.708 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:16.709 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:16.709 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:16 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:16.219+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:17.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:17.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:17.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:17.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:17.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:17.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:16 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:16.935+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:17.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:17.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:17.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:17.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:17.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:17.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:16 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:16.958+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:17.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:17.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:17.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:17.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:17.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:17.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:17 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:17.201+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:18.165 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:18.165 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:18.165 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:18.165 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:18.165 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:18.166 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:17.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:18.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:18.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:18.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:18.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:18.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:18.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:17 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:17.991+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:18.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:04.526254+0000 front 2024-04-23T18:28:04.526067+0000 (oldest deadline 2024-04-23T18:28:28.626238+0000) 2024-04-23T18:29:18.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:15.528544+0000 front 2024-04-23T18:28:15.528730+0000 (oldest deadline 2024-04-23T18:28:36.628578+0000) 2024-04-23T18:29:18.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:18.830665+0000 front 2024-04-23T18:28:18.830553+0000 (oldest deadline 2024-04-23T18:28:44.631195+0000) 2024-04-23T18:29:18.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:28.732073+0000 front 2024-04-23T18:28:28.732166+0000 (oldest deadline 2024-04-23T18:28:54.632084+0000) 2024-04-23T18:29:18.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:39.332975+0000 front 2024-04-23T18:28:39.332992+0000 (oldest deadline 2024-04-23T18:29:00.432882+0000) 2024-04-23T18:29:18.458 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:18 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9[61790]: 2024-04-23T18:29:18.177+0000 7fdfd53a8700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:43.234430+0000 front 2024-04-23T18:28:43.234317+0000 (oldest deadline 2024-04-23T18:29:09.134415+0000) 2024-04-23T18:29:19.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.885+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:19.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.885+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:19.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.886+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:19.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.886+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:19.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.886+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:19.243 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:18 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:18.886+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:19.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:19.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:19.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:19.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:19.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:19.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.025+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:19.708 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:19 smithi026 podman[90564]: 2024-04-23 18:29:19.465050132 +0000 UTC m=+5.451476063 container died 7612ab22c2b8c2cfd18911a0442ca8cb70065b7dad9d2dbd4f616433b4a2a93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-23T18:29:20.145 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:19 smithi026 podman[90564]: 2024-04-23 18:29:19.81931491 +0000 UTC m=+5.805740888 container cleanup 7612ab22c2b8c2cfd18911a0442ca8cb70065b7dad9d2dbd4f616433b4a2a93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:29:20.146 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:19 smithi026 bash[90564]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:20.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:19 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:19.920+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:20.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:20.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:19 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:19.975+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:20.457 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90581]: 2024-04-23 18:29:20.153846054 +0000 UTC m=+0.683693090 container remove 7612ab22c2b8c2cfd18911a0442ca8cb70065b7dad9d2dbd4f616433b4a2a93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:29:20.798 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90671]: 2024-04-23 18:29:20.403137244 +0000 UTC m=+0.018355678 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:20.799 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90671]: 2024-04-23 18:29:20.521366198 +0000 UTC m=+0.136584613 container create b974c472c418508f3f64956b58328584d792030508342155fee3b10b27539e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T18:29:20.799 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90671]: 2024-04-23 18:29:20.723550413 +0000 UTC m=+0.338768827 container init b974c472c418508f3f64956b58328584d792030508342155fee3b10b27539e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T18:29:20.799 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90671]: 2024-04-23 18:29:20.726742933 +0000 UTC m=+0.341961351 container start b974c472c418508f3f64956b58328584d792030508342155fee3b10b27539e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T18:29:20.799 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:20 smithi026 podman[90671]: 2024-04-23 18:29:20.806670801 +0000 UTC m=+0.421889216 container attach b974c472c418508f3f64956b58328584d792030508342155fee3b10b27539e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:29:21.208 INFO:journalctl@ceph.osd.9.smithi026.stdout:Apr 23 18:29:21 smithi026 podman[90671]: 2024-04-23 18:29:21.030649963 +0000 UTC m=+0.645868368 container died b974c472c418508f3f64956b58328584d792030508342155fee3b10b27539e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:21.240 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:21.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:21.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:21.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:21.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:21.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:20 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:20.915+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:21.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:20 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:20.996+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:21.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T18:29:21.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:21.641 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T18:29:21.641 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T18:29:21.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.10 2024-04-23T18:29:21.934 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 systemd[1]: Stopping Ceph osd.10 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:21.934 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.940+0000 7fc3c074d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:22.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.940+0000 7fc3c074d700 -1 osd.10 101 *** Got signal Terminated *** 2024-04-23T18:29:22.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.940+0000 7fc3c074d700 -1 osd.10 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:29:22.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:22.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:22.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:22.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:22.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:22.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:22.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:21 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:21.953+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:22.240 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:22.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:22.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:22.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:22.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:22.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:21.925+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:23.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:23.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:23.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:23.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:23.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:23.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:22 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:22.922+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:23.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:23.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:23.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:23.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:23.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:23.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:23.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:22 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:22.985+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:24.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:24.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:24.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:24.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:24.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:24.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:24.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:23 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:23.944+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:24.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:24.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:24.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:24.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:24.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:24.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:23 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:23.875+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:25.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:25.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:25.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:25.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:25.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:25.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:25.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:24 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:24.914+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:25.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:24 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:24.851+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:26.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:26.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:25.854+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:26.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:26.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:26.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:26.441 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:26.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:26.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:26.442 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:25 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:25.961+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:27.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.250064+0000 front 2024-04-23T18:28:06.250037+0000 (oldest deadline 2024-04-23T18:28:29.150176+0000) 2024-04-23T18:29:27.191 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:12.651255+0000 front 2024-04-23T18:28:12.651309+0000 (oldest deadline 2024-04-23T18:28:37.351426+0000) 2024-04-23T18:29:27.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:21.452858+0000 front 2024-04-23T18:28:21.452792+0000 (oldest deadline 2024-04-23T18:28:47.352580+0000) 2024-04-23T18:29:27.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:31.453684+0000 front 2024-04-23T18:28:31.453666+0000 (oldest deadline 2024-04-23T18:28:57.353605+0000) 2024-04-23T18:29:27.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.354103+0000 front 2024-04-23T18:28:37.354084+0000 (oldest deadline 2024-04-23T18:29:01.454166+0000) 2024-04-23T18:29:27.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:47.557028+0000 front 2024-04-23T18:28:47.556913+0000 (oldest deadline 2024-04-23T18:29:13.456843+0000) 2024-04-23T18:29:27.192 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:26 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10[57372]: 2024-04-23T18:29:26.912+0000 7fc3bcd41700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.26:6822 osd.6 since back 2024-04-23T18:28:53.457425+0000 front 2024-04-23T18:28:53.457475+0000 (oldest deadline 2024-04-23T18:29:19.357283+0000) 2024-04-23T18:29:27.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:27.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:27.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:27.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:27.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:27.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:27.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:26 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:26.892+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:27.683 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:27 smithi110 podman[69681]: 2024-04-23 18:29:27.396075736 +0000 UTC m=+5.560890917 container died eab00604fb0e9cfa4fe92c175a4cc3157c6137a01bafbf1c2c02462d3b33c290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:29:27.683 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:27 smithi110 podman[69681]: 2024-04-23 18:29:27.691139876 +0000 UTC m=+5.855955052 container cleanup eab00604fb0e9cfa4fe92c175a4cc3157c6137a01bafbf1c2c02462d3b33c290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T18:29:27.941 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:27 smithi110 bash[69681]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10 2024-04-23T18:29:27.941 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:27 smithi110 podman[69700]: 2024-04-23 18:29:27.925456385 +0000 UTC m=+0.524319918 container remove eab00604fb0e9cfa4fe92c175a4cc3157c6137a01bafbf1c2c02462d3b33c290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:28.196 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:28.197 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:27 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:27.842+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:28.503 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.217803733 +0000 UTC m=+0.109412242 container create 3693c8686ba1794079d1d9ad8a757a6b0f4715135a2f2ae1949899b431009759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:28.504 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.126441465 +0000 UTC m=+0.018049975 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:28.504 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.436779069 +0000 UTC m=+0.328387579 container init 3693c8686ba1794079d1d9ad8a757a6b0f4715135a2f2ae1949899b431009759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-23T18:29:28.504 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.439929365 +0000 UTC m=+0.331537876 container start 3693c8686ba1794079d1d9ad8a757a6b0f4715135a2f2ae1949899b431009759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:28.504 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.511580848 +0000 UTC m=+0.403189376 container attach 3693c8686ba1794079d1d9ad8a757a6b0f4715135a2f2ae1949899b431009759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T18:29:28.941 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 23 18:29:28 smithi110 podman[69790]: 2024-04-23 18:29:28.735680424 +0000 UTC m=+0.627288939 container died 3693c8686ba1794079d1d9ad8a757a6b0f4715135a2f2ae1949899b431009759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:29:29.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:29.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:29.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:29.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:29.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:29.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:29.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:28 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:28.835+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:29.314 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T18:29:29.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:29.351 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T18:29:29.351 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T18:29:29.351 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.11 2024-04-23T18:29:29.645 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 systemd[1]: Stopping Ceph osd.11 for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:29.645 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.651+0000 7f081be77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:29.645 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.652+0000 7f081be77700 -1 osd.11 101 *** Got signal Terminated *** 2024-04-23T18:29:29.645 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.652+0000 7f081be77700 -1 osd.11 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:29:29.990 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:29.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:29 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:29.865+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:31.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:31.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:31.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:31.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:31.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:31.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:31.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:30 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:30.879+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:32.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:32.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:32.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:32.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:32.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:32.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:32.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:31 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:31.910+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:33.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:33.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:33.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:33.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:33.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:33.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:33.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:32 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:32.959+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-04-23T18:28:06.072053+0000 front 2024-04-23T18:28:06.071854+0000 (oldest deadline 2024-04-23T18:28:29.571841+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-23T18:28:14.173511+0000 front 2024-04-23T18:28:14.173403+0000 (oldest deadline 2024-04-23T18:28:40.073510+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2024-04-23T18:28:20.074155+0000 front 2024-04-23T18:28:20.074131+0000 (oldest deadline 2024-04-23T18:28:46.475392+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.26:6814 osd.3 since back 2024-04-23T18:28:26.476051+0000 front 2024-04-23T18:28:26.476040+0000 (oldest deadline 2024-04-23T18:28:57.077016+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-23T18:28:37.077567+0000 front 2024-04-23T18:28:37.077593+0000 (oldest deadline 2024-04-23T18:29:01.777440+0000) 2024-04-23T18:29:34.241 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6812 osd.5 since back 2024-04-23T18:28:44.078339+0000 front 2024-04-23T18:28:44.078317+0000 (oldest deadline 2024-04-23T18:29:08.778416+0000) 2024-04-23T18:29:34.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-23T18:29:02.780670+0000 front 2024-04-23T18:29:02.780581+0000 (oldest deadline 2024-04-23T18:29:24.480439+0000) 2024-04-23T18:29:34.242 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:33 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11[58302]: 2024-04-23T18:29:33.969+0000 7f0814463700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.131:6820 osd.8 since back 2024-04-23T18:29:09.081945+0000 front 2024-04-23T18:29:09.081982+0000 (oldest deadline 2024-04-23T18:29:33.682384+0000) 2024-04-23T18:29:35.491 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:35 smithi131 podman[70898]: 2024-04-23 18:29:35.202525984 +0000 UTC m=+5.647140629 container died a866c9c0eb06a85ac341112fdf5666e42c8ac9eedde7dcceafc7c9f5ea094b73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:29:35.990 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:35 smithi131 podman[70898]: 2024-04-23 18:29:35.527186418 +0000 UTC m=+5.971801071 container cleanup a866c9c0eb06a85ac341112fdf5666e42c8ac9eedde7dcceafc7c9f5ea094b73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T18:29:35.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:35 smithi131 bash[70898]: ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11 2024-04-23T18:29:35.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:35 smithi131 podman[70915]: 2024-04-23 18:29:35.669883345 +0000 UTC m=+0.461813957 container remove a866c9c0eb06a85ac341112fdf5666e42c8ac9eedde7dcceafc7c9f5ea094b73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-23T18:29:36.348 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:35.982807342 +0000 UTC m=+0.017612220 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:29:36.348 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:36.095971131 +0000 UTC m=+0.130776061 container create c8d9180d8a508b822f6d2f2144023f80f9e133c8890e9902f023b1f0a0d822d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T18:29:36.349 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:36.356601642 +0000 UTC m=+0.391406496 container init c8d9180d8a508b822f6d2f2144023f80f9e133c8890e9902f023b1f0a0d822d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:29:36.646 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:36.359797025 +0000 UTC m=+0.394601882 container start c8d9180d8a508b822f6d2f2144023f80f9e133c8890e9902f023b1f0a0d822d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T18:29:36.646 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:36.439929431 +0000 UTC m=+0.474734301 container attach c8d9180d8a508b822f6d2f2144023f80f9e133c8890e9902f023b1f0a0d822d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T18:29:36.991 INFO:journalctl@ceph.osd.11.smithi131.stdout:Apr 23 18:29:36 smithi131 podman[71007]: 2024-04-23 18:29:36.654353853 +0000 UTC m=+0.689158701 container died c8d9180d8a508b822f6d2f2144023f80f9e133c8890e9902f023b1f0a0d822d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T18:29:37.333 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T18:29:37.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:37.373 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T18:29:37.373 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-23T18:29:37.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.a 2024-04-23T18:29:37.679 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:37 smithi026 systemd[1]: Stopping Ceph mds.a for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:37.957 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a[68356]: 2024-04-23T18:29:37.685+0000 7f3cc5e4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:37.958 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:37 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a[68356]: 2024-04-23T18:29:37.685+0000 7f3cc5e4d700 -1 mds.a *** got signal Terminated *** 2024-04-23T18:29:41.957 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:41 smithi026 podman[90787]: 2024-04-23 18:29:41.700037231 +0000 UTC m=+4.120521689 container died eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T18:29:42.320 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:42 smithi026 podman[90787]: 2024-04-23 18:29:42.069188507 +0000 UTC m=+4.489672972 container cleanup eaedc3689afc8b6d491d8f584ceee58e2255da4cef79e4cb99abbacb77bd26bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-23T18:29:42.320 INFO:journalctl@ceph.mds.a.smithi026.stdout:Apr 23 18:29:42 smithi026 bash[90787]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-a 2024-04-23T18:29:42.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.a.service' 2024-04-23T18:29:42.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:42.529 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-23T18:29:42.530 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-23T18:29:42.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.d 2024-04-23T18:29:42.856 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:42 smithi026 systemd[1]: Stopping Ceph mds.d for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:43.207 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d[67395]: 2024-04-23T18:29:42.864+0000 7f67c8704700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:43.208 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:42 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d[67395]: 2024-04-23T18:29:42.864+0000 7f67c8704700 -1 mds.d *** got signal Terminated *** 2024-04-23T18:29:46.958 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:46 smithi026 podman[90907]: 2024-04-23 18:29:46.701203853 +0000 UTC m=+3.941647755 container died e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T18:29:47.457 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:47 smithi026 podman[90907]: 2024-04-23 18:29:47.047778062 +0000 UTC m=+4.288221957 container cleanup e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:47.458 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:47 smithi026 bash[90907]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d 2024-04-23T18:29:47.458 INFO:journalctl@ceph.mds.d.smithi026.stdout:Apr 23 18:29:47 smithi026 podman[90933]: 2024-04-23 18:29:47.26557089 +0000 UTC m=+0.559643867 container remove e73333c2971789d316b794ca99ce4fb6b5bac181ab729e9730f15cfac419a205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T18:29:47.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.d.service' 2024-04-23T18:29:47.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:47.526 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-23T18:29:47.526 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-23T18:29:47.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.g 2024-04-23T18:29:47.852 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:47 smithi026 systemd[1]: Stopping Ceph mds.g for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:48.208 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-g[67988]: 2024-04-23T18:29:47.859+0000 7f5047fa1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:48.208 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:47 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-g[67988]: 2024-04-23T18:29:47.859+0000 7f5047fa1700 -1 mds.g *** got signal Terminated *** 2024-04-23T18:29:51.361 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:51 smithi026 podman[91042]: 2024-04-23 18:29:51.091956772 +0000 UTC m=+3.335359548 container died 91126b77f56ba2c0a88861f94032b74fd5fe82b3cce9bdc1a5de1a1a64c7e56e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:29:51.707 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:51 smithi026 podman[91042]: 2024-04-23 18:29:51.467949003 +0000 UTC m=+3.711351823 container cleanup 91126b77f56ba2c0a88861f94032b74fd5fe82b3cce9bdc1a5de1a1a64c7e56e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:29:51.708 INFO:journalctl@ceph.mds.g.smithi026.stdout:Apr 23 18:29:51 smithi026 bash[91042]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-g 2024-04-23T18:29:51.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.g.service' 2024-04-23T18:29:51.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:51.939 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-23T18:29:51.940 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-23T18:29:51.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.j 2024-04-23T18:29:52.297 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:52 smithi026 systemd[1]: Stopping Ceph mds.j for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:52.707 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j[67017]: 2024-04-23T18:29:52.304+0000 7f205f713700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:52.708 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:52 smithi026 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j[67017]: 2024-04-23T18:29:52.304+0000 7f205f713700 -1 mds.j *** got signal Terminated *** 2024-04-23T18:29:56.188 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:55 smithi026 podman[91164]: 2024-04-23 18:29:55.847017592 +0000 UTC m=+3.672867876 container died 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T18:29:56.188 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:56 smithi026 podman[91164]: 2024-04-23 18:29:56.195929007 +0000 UTC m=+4.021779292 container cleanup 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:29:56.457 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:56 smithi026 bash[91164]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j 2024-04-23T18:29:56.457 INFO:journalctl@ceph.mds.j.smithi026.stdout:Apr 23 18:29:56 smithi026 podman[91178]: 2024-04-23 18:29:56.371962654 +0000 UTC m=+0.522805951 container remove 3e4e9a7b19741c94a273f341a0f8f1340d3c5a31933f461392bf00ec3729c2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:29:56.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.j.service' 2024-04-23T18:29:56.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:29:56.655 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-23T18:29:56.655 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-23T18:29:56.656 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.b 2024-04-23T18:29:56.941 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:29:56 smithi110 systemd[1]: Stopping Ceph mds.b for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:29:57.441 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:29:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: 2024-04-23T18:29:57.040+0000 7f8e6ce6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:29:57.441 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:29:57 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b[63036]: 2024-04-23T18:29:57.040+0000 7f8e6ce6e700 -1 mds.b *** got signal Terminated *** 2024-04-23T18:30:00.624 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:30:00 smithi110 podman[69907]: 2024-04-23 18:30:00.338420088 +0000 UTC m=+3.479338084 container died 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T18:30:00.941 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:30:00 smithi110 podman[69907]: 2024-04-23 18:30:00.7152361 +0000 UTC m=+3.856154139 container cleanup 2e93d24b46a6a33f7964543531501bbbc3b51fec3c643adb623dc378adf16fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:30:00.941 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 23 18:30:00 smithi110 bash[69907]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-b 2024-04-23T18:30:01.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.b.service' 2024-04-23T18:30:01.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:01.120 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-23T18:30:01.121 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-23T18:30:01.121 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.e 2024-04-23T18:30:01.441 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:01 smithi110 systemd[1]: Stopping Ceph mds.e for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:01.941 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: 2024-04-23T18:30:01.451+0000 7f3bdf367700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:01.941 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:01 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e[62409]: 2024-04-23T18:30:01.451+0000 7f3bdf367700 -1 mds.e *** got signal Terminated *** 2024-04-23T18:30:04.691 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:04 smithi110 podman[70031]: 2024-04-23 18:30:04.342261454 +0000 UTC m=+2.990693505 container died 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:30:05.036 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:04 smithi110 podman[70031]: 2024-04-23 18:30:04.818666771 +0000 UTC m=+3.467098825 container cleanup 76fdb3eaaa20697d4a9a11e7676d477cef3026bb47df59d852c64e407f3f6ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T18:30:05.036 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 23 18:30:04 smithi110 bash[70031]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-e 2024-04-23T18:30:05.164 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.e.service' 2024-04-23T18:30:05.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:05.198 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-23T18:30:05.198 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-23T18:30:05.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.h 2024-04-23T18:30:05.441 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:05 smithi110 systemd[1]: Stopping Ceph mds.h for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:05.941 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: 2024-04-23T18:30:05.539+0000 7f0bcc60c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:05.941 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:05 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h[62715]: 2024-04-23T18:30:05.539+0000 7f0bcc60c700 -1 mds.h *** got signal Terminated *** 2024-04-23T18:30:08.631 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:08 smithi110 podman[70163]: 2024-04-23 18:30:08.347345037 +0000 UTC m=+2.923642815 container died 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:30:08.941 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:08 smithi110 podman[70163]: 2024-04-23 18:30:08.722430593 +0000 UTC m=+3.298728410 container cleanup 709e020b41debffb8727bbba675b97aa7d6bcd543dcddc01c51bb66704c9ee03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T18:30:08.941 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 23 18:30:08 smithi110 bash[70163]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-h 2024-04-23T18:30:09.143 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.h.service' 2024-04-23T18:30:09.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:09.175 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-23T18:30:09.175 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-23T18:30:09.176 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.k 2024-04-23T18:30:09.501 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:09 smithi110 systemd[1]: Stopping Ceph mds.k for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:09.941 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:09 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k[62089]: 2024-04-23T18:30:09.508+0000 7fbae3ced700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:09.941 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:09 smithi110 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k[62089]: 2024-04-23T18:30:09.508+0000 7fbae3ced700 -1 mds.k *** got signal Terminated *** 2024-04-23T18:30:13.191 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:12 smithi110 podman[70285]: 2024-04-23 18:30:12.948193251 +0000 UTC m=+3.548073832 container died 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-23T18:30:13.660 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 podman[70285]: 2024-04-23 18:30:13.484208661 +0000 UTC m=+4.084089230 container cleanup 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:30:13.660 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 bash[70285]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k 2024-04-23T18:30:13.896 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.k.service' 2024-04-23T18:30:13.915 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 podman[70299]: 2024-04-23 18:30:13.668330689 +0000 UTC m=+0.714291857 container remove 68c56aee4ffcf32b90acb0ba3d8b91f77c2b3510a91070b703286b60939ff790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:30:13.916 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-23T18:30:13.916 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 systemd[1]: Stopped Ceph mds.k for 2a701120-019c-11ef-bc93-c7b262605968. 2024-04-23T18:30:13.916 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 23 18:30:13 smithi110 systemd[1]: ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.034s CPU time. 2024-04-23T18:30:13.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:13.927 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-23T18:30:13.927 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-23T18:30:13.927 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c 2024-04-23T18:30:14.235 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:13 smithi131 systemd[1]: Stopping Ceph mds.c for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:14.491 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: 2024-04-23T18:30:14.242+0000 7fba5f776700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:14.491 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:14 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c[66883]: 2024-04-23T18:30:14.242+0000 7fba5f776700 -1 mds.c *** got signal Terminated *** 2024-04-23T18:30:16.585 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:16 smithi131 podman[71126]: 2024-04-23 18:30:16.3426016 +0000 UTC m=+2.212254036 container died 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:30:16.585 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:16 smithi131 podman[71126]: 2024-04-23 18:30:16.592758392 +0000 UTC m=+2.462410836 container cleanup 91cabb01121f4f97f79cd6dcf7aea920c2561c6bcc186582b75d85edf33ce45b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T18:30:16.844 INFO:journalctl@ceph.mds.c.smithi131.stdout:Apr 23 18:30:16 smithi131 bash[71126]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-c 2024-04-23T18:30:16.963 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.c.service' 2024-04-23T18:30:16.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:16.999 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-23T18:30:16.999 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-23T18:30:16.999 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.f 2024-04-23T18:30:17.240 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:17 smithi131 systemd[1]: Stopping Ceph mds.f for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:17.741 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: 2024-04-23T18:30:17.346+0000 7f83b1b92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:17.741 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:17 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f[63131]: 2024-04-23T18:30:17.346+0000 7f83b1b92700 -1 mds.f *** got signal Terminated *** 2024-04-23T18:30:20.663 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:20 smithi131 podman[71246]: 2024-04-23 18:30:20.349540839 +0000 UTC m=+3.126203296 container died e706f4ff5f2b06c85d90afa5eb08525ef16b29b000f9086112547e2c6e13ed82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T18:30:20.664 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:20 smithi131 podman[71246]: 2024-04-23 18:30:20.671882528 +0000 UTC m=+3.448544987 container cleanup e706f4ff5f2b06c85d90afa5eb08525ef16b29b000f9086112547e2c6e13ed82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T18:30:20.991 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:20 smithi131 bash[71246]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f 2024-04-23T18:30:20.991 INFO:journalctl@ceph.mds.f.smithi131.stdout:Apr 23 18:30:20 smithi131 podman[71260]: 2024-04-23 18:30:20.89130691 +0000 UTC m=+0.545617406 container remove e706f4ff5f2b06c85d90afa5eb08525ef16b29b000f9086112547e2c6e13ed82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:30:21.137 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.f.service' 2024-04-23T18:30:21.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:21.169 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-23T18:30:21.169 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-23T18:30:21.169 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.i 2024-04-23T18:30:21.476 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:21 smithi131 systemd[1]: Stopping Ceph mds.i for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:21.740 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: 2024-04-23T18:30:21.483+0000 7fbe72a7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:21.741 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:21 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i[63753]: 2024-04-23T18:30:21.483+0000 7fbe72a7c700 -1 mds.i *** got signal Terminated *** 2024-04-23T18:30:24.642 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:24 smithi131 podman[71368]: 2024-04-23 18:30:24.348150117 +0000 UTC m=+2.963117995 container died a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T18:30:24.642 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:24 smithi131 podman[71368]: 2024-04-23 18:30:24.650791154 +0000 UTC m=+3.265759040 container cleanup a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-23T18:30:24.990 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:24 smithi131 bash[71368]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i 2024-04-23T18:30:24.991 INFO:journalctl@ceph.mds.i.smithi131.stdout:Apr 23 18:30:24 smithi131 podman[71382]: 2024-04-23 18:30:24.801806267 +0000 UTC m=+0.453787450 container remove a8c621aae573b58eb2b946ae00c7ff03418543a1bbffc6f435a2fc9a0197760e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:30:25.005 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.i.service' 2024-04-23T18:30:25.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:25.036 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-23T18:30:25.036 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-23T18:30:25.036 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.l 2024-04-23T18:30:25.354 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:25 smithi131 systemd[1]: Stopping Ceph mds.l for 2a701120-019c-11ef-bc93-c7b262605968... 2024-04-23T18:30:25.740 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-l[62817]: 2024-04-23T18:30:25.362+0000 7f3caf036700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:30:25.741 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:25 smithi131 ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-l[62817]: 2024-04-23T18:30:25.362+0000 7f3caf036700 -1 mds.l *** got signal Terminated *** 2024-04-23T18:30:29.656 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:29 smithi131 podman[71488]: 2024-04-23 18:30:29.368734628 +0000 UTC m=+4.108432549 container died 1d1ff2f5866c775b1e4030b2a368a40ea91d91b0d03a2ac16573296c3c9ba679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:30:29.990 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:29 smithi131 podman[71488]: 2024-04-23 18:30:29.762990618 +0000 UTC m=+4.502688549 container cleanup 1d1ff2f5866c775b1e4030b2a368a40ea91d91b0d03a2ac16573296c3c9ba679 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:30:29.991 INFO:journalctl@ceph.mds.l.smithi131.stdout:Apr 23 18:30:29 smithi131 bash[71488]: ceph-2a701120-019c-11ef-bc93-c7b262605968-mds-l 2024-04-23T18:30:30.131 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a701120-019c-11ef-bc93-c7b262605968@mds.l.service' 2024-04-23T18:30:30.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:30:30.161 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-23T18:30:30.162 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:30:30.363 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:30:32.839 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:30:33.026 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:30:35.469 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:30:35.658 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:30:37.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:30:37.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:30:37.955 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:30:37.981 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T18:30:37.982 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi026/crash 2024-04-23T18:30:37.983 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:30:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:tar: /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:30:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:30:38.025 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi110/crash 2024-04-23T18:30:38.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:30:38.124 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:30:38.124 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:30:38.126 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi131/crash 2024-04-23T18:30:38.127 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:30:38.159 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/2a701120-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:30:38.159 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:30:38.160 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T18:30:38.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-23T18:30:38.199 INFO:tasks.cephadm:Compressing logs... 2024-04-23T18:30:38.200 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:30:38.243 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:30:38.245 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:30:38.267 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:30:38.267 INFO:teuthology.orchestra.run.smithi026.stderr:: No such file or directory 2024-04-23T18:30:38.267 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:30:38.274 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:30:38.274 INFO:teuthology.orchestra.run.smithi131.stderr:: No such file or directory 2024-04-23T18:30:38.274 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T18:30:38.274 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.c.log: -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:30:38.275 INFO:teuthology.orchestra.run.smithi131.stderr: 90.4% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:30:38.286 INFO:teuthology.orchestra.run.smithi131.stderr: 91.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-23T18:30:38.286 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:30:38.287 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T18:30:38.287 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T18:30:38.287 INFO:teuthology.orchestra.run.smithi131.stderr: 83.7% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:30:38.289 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:30:38.289 INFO:teuthology.orchestra.run.smithi110.stderr:: No such file or directory 2024-04-23T18:30:38.289 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:30:38.289 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T18:30:38.289 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:30:38.290 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr: 91.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:30:38.291 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T18:30:38.292 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.a.log: 86.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 90.3% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.6.log: 84.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:30:38.293 INFO:teuthology.orchestra.run.smithi026.stderr: 94.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log 94.6% 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr: 94.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:30:38.294 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:30:38.295 INFO:teuthology.orchestra.run.smithi026.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:30:38.296 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:30:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-23T18:30:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:30:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:30:38.307 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-23T18:30:38.321 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:30:38.322 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:30:38.322 INFO:teuthology.orchestra.run.smithi110.stderr: 94.5% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-23T18:30:38.335 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:30:38.336 INFO:teuthology.orchestra.run.smithi131.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:30:38.339 INFO:teuthology.orchestra.run.smithi131.stderr: 94.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-23T18:30:38.632 INFO:teuthology.orchestra.run.smithi026.stderr: 90.1% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T18:30:38.708 INFO:teuthology.orchestra.run.smithi131.stderr: 91.8% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T18:30:38.774 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T18:30:39.291 INFO:teuthology.orchestra.run.smithi026.stderr: 91.2% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T18:30:40.014 INFO:teuthology.orchestra.run.smithi131.stderr: 95.3% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-23T18:30:40.837 INFO:teuthology.orchestra.run.smithi131.stderr: 95.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-23T18:30:41.269 INFO:teuthology.orchestra.run.smithi026.stderr: 93.8% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T18:30:41.497 INFO:teuthology.orchestra.run.smithi026.stderr: 94.2% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T18:30:41.641 INFO:teuthology.orchestra.run.smithi110.stderr: 95.8% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-23T18:30:42.027 INFO:teuthology.orchestra.run.smithi110.stderr: 93.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T18:30:42.203 INFO:teuthology.orchestra.run.smithi026.stderr: 94.2% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T18:30:42.423 INFO:teuthology.orchestra.run.smithi131.stderr: 94.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T18:30:42.472 INFO:teuthology.orchestra.run.smithi131.stderr: 94.1% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T18:30:42.492 INFO:teuthology.orchestra.run.smithi026.stderr: 94.0% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T18:30:42.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-23T18:30:42.494 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m4.237s 2024-04-23T18:30:42.494 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m15.261s 2024-04-23T18:30:42.494 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.543s 2024-04-23T18:30:42.699 INFO:teuthology.orchestra.run.smithi131.stderr: 94.3% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T18:30:42.751 INFO:teuthology.orchestra.run.smithi110.stderr: 94.1% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T18:30:42.779 INFO:teuthology.orchestra.run.smithi110.stderr: 94.4% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T18:30:42.962 INFO:teuthology.orchestra.run.smithi131.stderr: 94.3% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T18:30:43.931 INFO:teuthology.orchestra.run.smithi110.stderr: 94.3% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T18:31:04.260 INFO:teuthology.orchestra.run.smithi110.stderr: 93.9% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-23T18:31:13.347 INFO:teuthology.orchestra.run.smithi131.stderr: 94.1% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-23T18:31:13.348 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-04-23T18:31:13.349 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m35.089s 2024-04-23T18:31:13.349 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m55.272s 2024-04-23T18:31:13.349 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m1.898s 2024-04-23T18:31:35.769 INFO:teuthology.orchestra.run.smithi110.stderr: 93.6% -- replaced with /var/log/ceph/2a701120-019c-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-23T18:31:35.775 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-23T18:31:35.775 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m57.511s 2024-04-23T18:31:35.775 INFO:teuthology.orchestra.run.smithi110.stderr:user 1m42.260s 2024-04-23T18:31:35.775 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m3.290s 2024-04-23T18:31:35.776 INFO:tasks.cephadm:Archiving logs... 2024-04-23T18:31:35.776 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi026/log 2024-04-23T18:31:35.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:31:37.828 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi110/log 2024-04-23T18:31:37.829 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:31:50.937 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi131/log 2024-04-23T18:31:50.938 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T18:31:57.772 INFO:tasks.cephadm:Removing cluster... 2024-04-23T18:31:57.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force 2024-04-23T18:31:57.976 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:31:58.249 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force 2024-04-23T18:31:58.438 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:31:58.880 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a701120-019c-11ef-bc93-c7b262605968 --force 2024-04-23T18:31:59.070 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: 2a701120-019c-11ef-bc93-c7b262605968 2024-04-23T18:31:59.376 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T18:31:59.376 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:31:59.396 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:31:59.412 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T18:31:59.427 INFO:tasks.cephadm:Teardown complete 2024-04-23T18:31:59.427 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T18:31:59.438 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T18:31:59.439 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:31:59.441 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:31:59.454 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T18:31:59.562 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T18:31:59.573 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T18:31:59.574 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:31:59.577 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:31:59.579 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:31:59.591 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:31:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:31:59.593 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:^- hv01.front.sepia.ceph.com 2 6 377 8 +1080us[+1080us] +/- 591ms 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -240us[ -236us] +/- 67ms 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +299us[ +299us] +/- 81ms 2024-04-23T18:31:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:31:59.598 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:31:59.598 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-04-23T18:31:59.598 INFO:teuthology.orchestra.run.smithi131.stdout:^- hv01.front.sepia.ceph.com 2 8 377 73 +1151us[+1151us] +/- 590ms 2024-04-23T18:31:59.598 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -229us[ -228us] +/- 67ms 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +318us[ +318us] +/- 81ms 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi026.stdout:^- hv01.front.sepia.ceph.com 2 6 377 5 +1059us[+1059us] +/- 591ms 2024-04-23T18:31:59.599 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -243us[ -229us] +/- 67ms 2024-04-23T18:31:59.600 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +303us[ +317us] +/- 81ms 2024-04-23T18:31:59.600 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:31:59.600 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T18:31:59.610 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T18:31:59.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T18:31:59.621 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:31:59.721 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:31:59.786 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T18:31:59.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:31:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:31:59.880 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:31:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:31:59.907 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:31:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:31:59.933 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T18:31:59.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T18:31:59.955 INFO:teuthology.task.internal:Duration was 1952.469540 seconds 2024-04-23T18:31:59.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T18:31:59.966 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T18:31:59.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:31:59.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:31:59.971 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T18:32:00.005 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:32:00.007 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:32:00.009 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:32:00.478 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T18:32:00.478 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-04-23T18:32:00.478 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:32:00.501 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-04-23T18:32:00.502 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:32:00.543 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-04-23T18:32:00.543 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T18:32:00.567 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T18:32:00.567 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:32:00.570 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:32:00.586 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T18:32:00.632 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T18:32:00.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:32:00.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:32:00.651 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T18:32:01.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T18:32:01.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T18:32:01.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:32:01.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:32:01.083 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T18:32:01.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T18:32:01.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.120 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.125 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.140 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-04-23T18:32:01.145 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-04-23T18:32:01.169 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-04-23T18:32:01.194 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:32:01.208 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:32:01.248 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T18:32:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:32:01.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T18:32:01.273 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T18:32:01.274 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi026 2024-04-23T18:32:01.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:32:01.315 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi110 2024-04-23T18:32:01.316 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:32:01.352 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100/remote/smithi131 2024-04-23T18:32:01.352 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T18:32:01.387 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T18:32:01.387 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:32:01.390 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:32:01.392 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T18:32:01.436 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T18:32:01.451 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T18:32:01.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T18:32:01.468 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T18:32:01.468 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:32:01.471 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:32:01.473 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T18:32:01.483 INFO:teuthology.orchestra.run.smithi026.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:32 /home/ubuntu/cephtest 2024-04-23T18:32:01.485 INFO:teuthology.orchestra.run.smithi110.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:32 /home/ubuntu/cephtest 2024-04-23T18:32:01.490 INFO:teuthology.orchestra.run.smithi131.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 18:32 /home/ubuntu/cephtest 2024-04-23T18:32:01.491 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T18:32:01.504 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T18:32:01.528 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T18:32:01.552 DEBUG:teuthology.nuke:shortname: smithi026 2024-04-23T18:32:01.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:32:01.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.432392', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2MeWAIPf5+honJbPW8q95+kLsIwe+CaCOIRqW2Y7Aod7GOU66CeYemrJfizi3EtDpwpcaT8Excx+djS4aY/9Q='} 2024-04-23T18:32:01.580 DEBUG:teuthology.nuke:shortname: smithi110 2024-04-23T18:32:01.580 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:32:01.608 DEBUG:teuthology.nuke:shortname: smithi131 2024-04-23T18:32:01.609 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:32:01.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.433806', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWASL7n+ZS0ePdXEdYuixB6KXomVNynUNGx5YgwJcztvQFIEm0VWH6L/xg0Oby2lZ28u1KTD8VD9LSNzCgR6tY='} 2024-04-23T18:32:01.635 INFO:teuthology.orchestra.console.smithi026:Power off 2024-04-23T18:32:01.636 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:32:01.648 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:54:01.435509', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEXOMv1VMbXuz8T+ca7x7Q46YAkwk+kwEoErjZOj4y2xWqW/D8IqQwSISRgiXJa0EQGmp7flPID8x+TQzcQayk='} 2024-04-23T18:32:01.659 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-04-23T18:32:01.660 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:01.681 INFO:teuthology.orchestra.console.smithi110:Power off 2024-04-23T18:32:01.682 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:32:01.691 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-23T18:32:01.703 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-04-23T18:32:01.704 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:01.719 INFO:teuthology.orchestra.console.smithi131:Power off 2024-04-23T18:32:01.719 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T18:32:01.729 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-04-23T18:32:01.745 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-04-23T18:32:01.745 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:01.770 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-23T18:32:05.692 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:05.730 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:05.771 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:05.803 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-23T18:32:05.842 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-04-23T18:32:05.882 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-04-23T18:32:09.806 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:09.843 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:09.884 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T18:32:09.916 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-04-23T18:32:09.952 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-04-23T18:32:09.993 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-04-23T18:32:10.017 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-04-23T18:32:10.053 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-04-23T18:32:10.094 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-04-23T18:32:10.201 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 1952.4695401191711 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi026 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=76ef22ae47c54a3e9262fe71a7b3b927 status: fail success: false 2024-04-23T18:32:10.202 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T18:32:10.283 INFO:teuthology.run:FAIL